mirror of
https://github.com/storytold/craft-anything.git
synced 2026-11-19 05:50:53 +01:00
1320 lines
98 KiB
Markdown
1320 lines
98 KiB
Markdown
# Pitfalls
|
||
|
||
Mistakes made while building this project, what they cost, and the rule that prevents each one.
|
||
Read this before your first change; add to it when you make a new mistake worth not repeating.
|
||
Keep entries short: what happened, then the rule.
|
||
|
||
## Git and pull requests
|
||
|
||
**Work began before its roadmap claim, on a branch without the required prefix.** Renaming the
|
||
branch and recording the claim during PR preparation fixes coordination going forward, not the
|
||
missed order. Create and publish the claim on a correctly named branch before implementation;
|
||
never describe a late claim as compliance with the claim-before-work rule.
|
||
|
||
**A commit landed on `main` without a PR.** `git switch main && git pull` was run with uncommitted
|
||
changes: the switch succeeded, the pull refused, and the failure was read as "still on my branch".
|
||
The next commit and `git push --force-with-lease` went to `main`.
|
||
- Check `git branch --show-current` before every commit and push.
|
||
- Push by name (`git push origin <branch>`), never a bare `git push` after switching around.
|
||
- Commit or stash before switching branches; read every line of a failed compound command.
|
||
- Maintainers: protect `main` (PRs required, squash only) so this can't happen.
|
||
|
||
**A PR merged with only its claim.** One shell command did commit, push, `gh pr create` and
|
||
`gh pr merge`; the pre-commit hook failed (it ran `cargo xtask secrets` on an older default Rust),
|
||
the `&&` chain stopped, but the next lines still ran, so #138 merged the claim commit under the
|
||
work's title. Run commit, push, create and merge as separate steps and read each result; run the
|
||
hook with the project's toolchain (`RUSTUP_TOOLCHAIN=1.99.0 git commit …`).
|
||
|
||
**A merge of commits no check ran on.** A script waited for one CI run of a PR, then merged
|
||
whatever the PR's head was by then; its author had pushed two more commits meanwhile, so #141
|
||
reached `main` with code neither CI nor review had seen. Merge with `--match-head-commit` set to
|
||
the commit the green run tested (`gh run view <id> --json headSha`), never to a head read later.
|
||
|
||
**Squash messages that lost their co-authors.** A batch of replays was merged from a script that
|
||
kept each message on one line and turned `\n` back into line breaks. A shell heredoc had collapsed
|
||
that step into one that did nothing, and `printf` broke other lines early, so a dozen squash
|
||
commits reached `main` with `\n\nCo-authored-by:` as plain text or with the line cut off, and
|
||
GitHub credited nobody. Write a squash message to a file with real line breaks, and read the first
|
||
commit a new merge script makes (`git log -1 --format=%B`) before it does a second.
|
||
|
||
**A squash commit names the wrong PR number.** The subject was written as "(#12)" before the PR
|
||
existed; it became #11. Take the number from `gh pr create`'s output (`${URL##*/}`) and use it.
|
||
|
||
**Two PRs pass alone and break `main` together.** One tightened `cargo xtask layers`, the other
|
||
added an app the new rule rejected; both were green on their own branch.
|
||
- Before merging a PR that was rebased before another merge, build and test the merged result:
|
||
`git merge-tree --write-tree origin/main origin/<branch>` for conflicts, then a temporary branch
|
||
with the merge, `cargo test --workspace`, clippy and `cargo xtask layers`.
|
||
- After merging several PRs in a row, run the full checks on `main` once more.
|
||
- It happened again with translations: #117 added UI strings and #115, merged right after it,
|
||
added a check that every string has a Chinese translation, so `cargo xtask ci` failed on
|
||
`main`. A PR that adds a check, or adds what a check covers, is tested on top of the other.
|
||
And again with a new catalog: #555 added "Pausing…" to every catalog there was, and #486 (Russian)
|
||
merged after it without it. A new language and a new UI string merged together need the string
|
||
in the new catalog too.
|
||
And with a layout change: #687 moved coding agents onto their own tab of what Detect found, and
|
||
#664, merged next, tested a button on that list without picking the tab. Checking that two PRs
|
||
merge cleanly isn't checking that their tests pass together: run the UI tests on the merge.
|
||
|
||
**A table row kept twice by a merge.** Two PRs edited the same STATUS.md row; resolving the
|
||
conflict kept both versions, so five rows reached `main` twice, each copy with a different
|
||
update, and the three OpenCode rows disagreed on whether SSH runs get the tools. When a
|
||
conflict is in a table row, merge the edits into one row, and after merging look for a row
|
||
name that appears twice.
|
||
|
||
**A dependency change merged without its lockfile.** Changes were staged with
|
||
`git add -A crates apps docs`, which leaves out the root `Cargo.lock`, so a new dependency reached
|
||
`main` without its lock entry. Stage by `git add -A` (then review `git status`), or list `Cargo.lock`
|
||
whenever a `Cargo.toml` changed.
|
||
|
||
**A superseded PR on someone else's branch.** Rebasing it means force-pushing their branch. Replay it
|
||
on a new branch, open a new PR that says it supersedes the old one, and comment on the old one.
|
||
Don't close or rewrite other people's work without asking.
|
||
|
||
**Review fixes pushed to a PR that had already merged.** #686 was merged hours before its review
|
||
comments were acted on; the fixes went to its branch and the threads were answered "Fixed", while
|
||
`main` had the code without them. Read `gh pr view --json state,headRefOid` before pushing to a PR's
|
||
branch, and land anything late in a new PR from `main`.
|
||
It happened again with the check in place: `gh pr view 1022 --json state -q .state && … git push`
|
||
printed `MERGED` and went on, because printing a state isn't failing. The check has to stop the
|
||
command: `test "$(gh pr view <n> --json state -q .state)" = OPEN && …`.
|
||
|
||
**A report checked against main, not the release it ran.** #736 (a build looping on an empty `app/`)
|
||
was read against main, where the stop after two unchanged turns already existed, so it looked fixed.
|
||
The reporter ran v0.3.0, which had neither that stop (#557) nor the empty-folder note (#749); both
|
||
merged the day after it shipped. Before calling a report fixed or still open, find the release it
|
||
names (`git merge-base --is-ancestor <fix> <release tag>`), and check the message text it quotes.
|
||
|
||
## Tests
|
||
|
||
**A TLS fixture closed before the handshake.** Accepted sockets inherited the listener's
|
||
nonblocking mode on Windows, so the fixture read `WouldBlock` and closed a valid connection.
|
||
Set the accepted socket to blocking mode before applying bounded I/O timeouts; consume the
|
||
complete request header before replying.
|
||
|
||
**A test module hid later production code from a policy gate.** The source scan stopped at the
|
||
first `cfg(test)`, and unreadable files looked empty. Inspect syntax for the actual test item,
|
||
keep checking its production siblings and source directories (even one named `tests`), and report source failures. `cfg(not(test))` and
|
||
`cfg(any(test, feature = "app"))` may contain production code; a cfg string containing "test"
|
||
isn't enough to exempt it. The source-policy checks follow explicit imports and module files,
|
||
not macro expansion, globs, re-exports or all Rust name resolution; compiler checks still matter.
|
||
Hidden module directories are source too. Symlinked source directories fail visibly rather than
|
||
being omitted or followed into cycles; ordinary Rust file links are read and checked. Git metadata
|
||
stays out of the directory walk. Module-level declarations explicitly referencing a file
|
||
include it even under an otherwise-exempt directory; discovery is cycle-safe and bounded to the
|
||
workspace. Missing cfg-dependent files still require compiler validation.
|
||
|
||
**A heartbeat sampled before shutdown raced a successful stop.** A child could write between
|
||
the baseline read and `stop()`, failing the descendant cleanup test even when cleanup worked.
|
||
Probe for exit instead of timing writes: the child holds a file lock the system releases only
|
||
when it exits, and the test takes it after the stop. Ignored child fixtures also need
|
||
an explicit launch guard so running ignored tests directly cannot write in the working directory.
|
||
The replacement's quoted `cmd /C` script still failed on Windows, and startup assertions skipped
|
||
cleanup. Use a guarded Rust parent instead of shell syntax, and stop before asserting on startup.
|
||
Its child must inherit the watched group/job: another `Process` or `Launch` creates a separate
|
||
Unix group and would no longer test the ancestor's shutdown.
|
||
|
||
**An isolated PATH still assumed system utilities lived in `/bin`.** The remote setup fixture
|
||
linked `mkdir`, `chmod` and `mv` there, so a host with utilities elsewhere got dangling links.
|
||
Resolve fixture utilities on the host, make their paths absolute, then populate the private PATH.
|
||
|
||
**Python compilation reused stale bytecode after a fast edit.** The new-project toolchain probe
|
||
changed valid Python to a syntax error within the same mtime second. `compileall` skipped it and
|
||
exited zero; the screenshot run then failed instead. Use `compileall -f` for a build check, and
|
||
hold the valid and broken fixture revisions at the same timestamp in the real-Python regression.
|
||
|
||
**A test started a real download on a contributor's machine.** A test called `engines_install` for
|
||
Unity; a later change made that download the vendor installer and open it on Windows. It also
|
||
created folders in the real config directory (`Session::default()` uses it).
|
||
- Tests never touch the network, never start installers or GUI programs, and never write outside a
|
||
temp folder. Use ids that are refused before any I/O, or local stand-in servers.
|
||
- When you widen what a tool does (a new download, a launch), grep the tests that call it.
|
||
- Tests build their session with `craft_anything_engine::testing::session()` (config, plugin
|
||
caches, engine store and projects in a temp folder); `cargo xtask layers` rejects
|
||
`Session::default()`. CLI tests set `CRAFT_ANYTHING_CONFIG_DIR` to a temp folder.
|
||
|
||
**A demo app wired to the person's settings.** The headless app the UI tests use built its AI hub
|
||
with `AiHub::new`, which saves to the real `config.toml` and the system keychain; no test had
|
||
added or removed an AI yet, so nothing noticed. Everything a test app can save goes to a temp
|
||
folder (`AiHub::with_stores`), not only the engine's session.
|
||
|
||
**A test that finds a program by Cargo's folder layout.** The CLI was looked for one folder up
|
||
from the test binary (`target/debug/deps`). CI follows stable Rust, and Rust 1.100's Cargo moved
|
||
test binaries to `target/debug/build/<package>/<hash>/out`, so `main` failed overnight with no
|
||
code change. When CI breaks on a commit that passed, compare the runs' toolchain lines first, and
|
||
don't rely on where Cargo keeps its intermediate files.
|
||
|
||
**A test that passes only where a program is installed.** A tool's `Requires::Engine` is checked
|
||
before it runs, so a test of its input checks passed on a machine with ilspycmd and failed on one
|
||
without. Point the test session's `engines.tools` path at a stand-in file when the tool refuses
|
||
before running anything, and run new tests on a machine without the program.
|
||
It happened again when Clean room began refusing tools before their program is looked for: a
|
||
"missing adb" test returned early on a machine with adb and failed in CI without it. A test of
|
||
what happens without a program builds its session in the mode the tool runs in.
|
||
It happened a third time with Ghidra: the test session's engine store was empty, but the lookup
|
||
still searched PATH, `~/.local/bin` and `/opt/ghidra`, so two Deep-mode tests got Ghidra's answer
|
||
on a machine that had it. `testing::session()` now sets `engines.only_configured`, so it finds only
|
||
the engines its settings or store name.
|
||
|
||
**A failing crate hid the next crate's tests.** `cargo test -p a -p b` stopped after a failure in
|
||
`a`, so `b`'s tests never ran; the rerun was filtered to the failing test, and a broken engine id
|
||
reached `main`. Run with `--no-fail-fast`, and after a fix rerun the whole crates, not a filter.
|
||
|
||
**A stopped server's port was no longer its identity.** A shutdown test connected to the old
|
||
port and treated any listener there as the stopped server, although another fixture could reuse it.
|
||
Observe the actual tool's registered server and a weak reference to its shared answerer; cancellation
|
||
must release that resource and empty its registry. Do not reserve and release a port before startup.
|
||
|
||
**A test that changes the whole process.** A UI test raised the process-wide stuck-lock banner; in
|
||
CI the banner moved the layout under another test running beside it, and its click missed. It passed
|
||
25 runs on a Mac. Anything a test sets for the whole process goes in a test program of its own.
|
||
|
||
**A released file lock that still looked held.** Saved sign-in reservations use `flock`; a
|
||
program another thread was starting held a copy of the lock file until its exec closed it, so a
|
||
sign-out right after the reservation was dropped was refused as busy in about one full engine
|
||
run in four, never alone. A non-blocking lock tries again for a moment (`lock_site`'s
|
||
`LOCK_GRACE`) before calling something busy, and a lock test is run inside the full suite.
|
||
|
||
**A test that passed only without the developer's `CARGO_TARGET_DIR`.** Since the program
|
||
lookup reads `CARGO_TARGET_DIR`, four pin tests failed on a machine that exported it: a live check
|
||
had left a real `snake` program in that shared folder, and the tests' project is also called snake.
|
||
The product had the same fault: any same-named program there was taken for the project's. Run the
|
||
engine tests with `CARGO_TARGET_DIR` set to a folder holding other builds, and a lookup in a
|
||
shared folder needs proof that a file is this project's, not just its name.
|
||
|
||
**A free port handed to another test.** For a negative TCP probe, a bound port stopped being
|
||
ours as soon as its listener was dropped. Another fixture could acquire it before the probe.
|
||
Use the unassigned loopback port zero to exercise a real connection refusal without borrowing
|
||
and releasing a shared ephemeral port.
|
||
|
||
**Fixtures that leak into each other.** A "nothing here" path sat beside a fake game folder in the
|
||
same temp directory, and game detection walked up and found the game. Give each case its own
|
||
`tempfile::tempdir()`.
|
||
|
||
**Tests that share a process-wide value.** The start guard keeps its mark's lock in a `static`;
|
||
one test's `opened()` emptied it while another test was between `begin` and its check, so
|
||
`main` failed now and then although the PR's own run had passed. Tests that touch the same
|
||
`static` take a shared test lock, even when each uses its own temp folder.
|
||
|
||
**A mock server closed on a partial request.** A loopback fixture read a request only once, then
|
||
replied and closed. TCP can split even a short header; unread bytes made that close reset the
|
||
connection. Consume the complete header before replying, with a size cap and an absolute deadline.
|
||
Negative probes use loopback port zero so another fixture cannot acquire a released port.
|
||
A controlled fragmented request reproduced the fixture defect; the old CI log did not preserve
|
||
the probe's underlying error, so it does not establish that failure's cause.
|
||
|
||
**A socket option set after the peer reset it.** A preview test set its clients' read timeout
|
||
after stopping the server. macOS answers `setsockopt` with `Invalid argument` on a socket closed in
|
||
both directions (a reset, or the peer's FIN after our own `shutdown(Write)`); Linux doesn't check,
|
||
so the test failed only on Macs. Set socket options right after connecting, before anything can
|
||
close the connection.
|
||
|
||
**A script written and run at once.** Tests that write a stand-in program and start it failed now
|
||
and then on Linux with "Text file busy": another test thread was starting a child at that moment,
|
||
and the child still held the new file open for writing. Starting a program goes through
|
||
`process::spawn_retrying`, which tries a busy file again for a moment; don't add a sleep to a test
|
||
instead.
|
||
|
||
**Proving a refactor changed nothing.** Unit tests that check substrings can miss changed prompts.
|
||
For a refactor that must keep output identical, dump the outputs for every combination before and
|
||
after and compare them byte for byte (done for every brief across 162 combinations in R1.1).
|
||
|
||
**A guard phrase that matched the wording it protects.** A test that no blind builder is told to
|
||
"look at the picture" failed on the new "you can't look at the pictures in reference/". Make a
|
||
forbidden phrase the instruction's exact words, check each one still appears in what the other
|
||
case is sent (or the guard goes stale without failing), and break the code once to see it fail.
|
||
|
||
**A Windows-only test that no check ran.** #748 added a second Antigravity folder to the search
|
||
path; the Windows test that counts those folders wasn't updated, and `main` stayed green because CI
|
||
runs on Ubuntu, where `cfg(windows)` tests don't compile. When a change touches a list a
|
||
`cfg(windows)` (or macOS) test checks, grep for it and run it on that system, or say in the PR that
|
||
you couldn't.
|
||
|
||
**A step moved to a worker, its test still counting frames.** #720 moved the picture size check
|
||
off the UI thread; a UI test that added pictures, ran three frames and then looked kept passing on
|
||
fast machines and failed in CI on unrelated PRs. When work moves to a worker, grep the tests that
|
||
call it and make them `wait` for the result, not `run_steps(n)`.
|
||
|
||
**A stand-in server that answers every request in order.** The AI tests' `mock` answered each
|
||
request with the next scripted reply, so when the agent began asking the server for its context
|
||
window before the first chat, that question took the chat's reply. A stand-in says which requests
|
||
it was scripted for and answers anything else as a real server that doesn't know would.
|
||
|
||
## Building and running
|
||
|
||
**A documentation edit rebuilt dependencies in a separate cache.** A sequential review follow-up
|
||
used another target directory for its commit hook, triggering an unnecessary build. Separate
|
||
build caches are for concurrent work; reuse the current checkout's cache for sequential work.
|
||
|
||
**A runner's PID isn't the window owner's.** `go run` starts a compiled child, so picturing the
|
||
runner missed the app (or a name fallback found another app). Search only the launched process
|
||
and its descendants, and read the UI tree from the same PID that supplied the picture.
|
||
|
||
**Runtime capture passed relative paths to a tool that needs absolute ones.** With a relative
|
||
project placement, the selected virtualenv interpreter and artifact paths stayed relative;
|
||
PATH lookup can return a relative path too. Resolve workspace paths against the process's
|
||
current directory, look up only bare program names, and make the result absolute without
|
||
dereferencing the virtualenv symlink. Test the generated capture request without opening a GUI.
|
||
The same rule covers every child started in app/: engine build arguments, option files, console
|
||
run checks, Play and pins must get absolute program, working-directory and path arguments; a
|
||
relative one is resolved against app/ again.
|
||
|
||
**A screenshot helper aborted inside CoreGraphics.** The first macOS screenshot probe used
|
||
ScreenCaptureKit from a command-line Swift helper without initializing AppKit's WindowServer
|
||
connection. An owned-window capture aborted in `CGS_REQUIRE_INIT`, while no-window regressions
|
||
passed. Initialize `NSApplication.shared` on the main actor before capture, and verify an actual
|
||
owned-window PNG as well as missing-window errors.
|
||
|
||
**A model catalogue queried in the wrong project.** OpenCode V2's raw `api get /api/model` does
|
||
not reliably select the invoking CLI's directory. Use the `model.list` operation with an explicit
|
||
`location[directory]` parameter, and test that the returned Location matches it. Over SSH,
|
||
expand the remote shell's working directory, never send a local project path.
|
||
|
||
**Rebuilding a binary that is running fails on Windows** ("Access is denied"), and the next command
|
||
silently used the old binary. Long-running binaries (live rebuilds, servers) get their own
|
||
`CARGO_TARGET_DIR`.
|
||
|
||
**A program started from a screen outlived the tests.** The Ghidra memory setting first asked
|
||
PowerShell for the computer's memory on a background thread while the screen drew; in UI tests
|
||
the test process ended first, the orphaned PowerShell kept the test runner's output pipe open,
|
||
and `cargo test | grep` never returned. Screens never start programs, not even "in the
|
||
background"; read a setting or a value a worker already has.
|
||
|
||
**Asking a process to stop isn't stopping it.** `RunningProcess::kill()` only raises a flag for the
|
||
supervisor thread; the window-capture tool returned and the CLI exited before the thread acted, so
|
||
the program it opened stayed open (and held the shell's pipe, so the command never returned). When
|
||
the caller may exit next, use `kill_and_wait`, and check after a live run that nothing was left
|
||
running.
|
||
|
||
**A turn's own process outlived its answer.** `run_cli` only sent `AgentEvent::TurnDone` once the
|
||
whole coding-agent subprocess exited, never once its JSON stream said the turn was done (the
|
||
`result` line). Claude Code can print that line and then linger (its own shutdown hook, logging)
|
||
for as long as the idle timeout allows — hours, for a builder turn — so a finished build stayed on
|
||
"Building"/"Pausing" long after the real work and even a final report were done, with no crash and
|
||
no error to find. Once the stream's own terminal event is parsed, stop waiting on the process
|
||
itself: fold a `result_seen` flag into `cancel_when` with a short grace period (`RESULT_GRACE`),
|
||
and don't report that as "Stopped." A terminal stream event, not the OS exit, is what ends a turn.
|
||
|
||
**A deadline that ended in a wait with none.** Every `Process` run has a time limit and a cancel,
|
||
but when either fired the supervisor called the library's `kill()`, which stops the program and
|
||
then waits for it with no limit (on Windows also for a job message the system doesn't promise).
|
||
A program that outlives its stop held the run, and Pause, for good (#1438, #981). A deadline covers
|
||
its own cleanup too: after a stop, poll for a moment and move on (`process::end`), and when a
|
||
library call says "kill", read whether it also waits. Not reproduced on the reporters' machines.
|
||
|
||
**A console held open by a program started next door.** On Windows Server 2022 opening a
|
||
pseudoconsole briefly makes inheritable handles (a Unix terminal too, until it's made close-on-exec),
|
||
so a program another thread started at that moment kept the console open until it ended, and closing
|
||
a console with programs still attached hung its host. Program starts and console or terminal opens
|
||
take `process::STARTING`; end a console's process tree before closing it.
|
||
|
||
**The newest program isn't the app.** Play started the newest executable in `target/`; a
|
||
workspace that builds an app and its CLI started the CLI, which printed help and quit with no
|
||
console, so the button did nothing, three times in a row. Pick what to start from what the project
|
||
declares (Cargo manifests: `default-run`, the package using a window library), never from file
|
||
dates, and make a start that exits at once say so.
|
||
|
||
**A launcher's exit read as the program's.** On some Windows machines `chrome.exe` hands the
|
||
browser to a second process and exits; the supervisor took that exit as the run ending and stopped
|
||
the whole group, so websites never opened ("closed after 0.3 s"). Where it didn't hand off, the
|
||
browser outlived the CLI (the job isn't killed when closed) and held the shell's output pipe, so
|
||
the command hung and old browsers piled up. Start programs that hand off with `.hands_off()`, and
|
||
after a live run check that the command returned and nothing it started is still running.
|
||
|
||
**A live browser test that died on a line about crashpad.** On Ubuntu with
|
||
`kernel.apparmor_restrict_unprivileged_userns=1`, a Chromium that has no AppArmor profile (Playwright's
|
||
copy, Chrome for Testing) can't build its sandbox and exits at once; the only stderr line we kept was a
|
||
harmless crashpad warning (`cpufreq/scaling_max_freq`), so all five live sign-in tests failed in 1.5 s with
|
||
"closed before it was ready". Don't turn the sandbox off in the product for it: point
|
||
`CRAFT_ANYTHING_TEST_CHROMIUM` at a throwaway wrapper in `/tmp` that adds `--no-sandbox`, only for a browser
|
||
that visits a server on 127.0.0.1. A helper that connects the moment the debugging port file appears must
|
||
also wait for the first page target (`first_page`): it raced the browser's own start.
|
||
|
||
**A separate config folder hides the agent's own settings.** To keep a live run from touching the
|
||
real `config.toml`, `XDG_CONFIG_HOME` was pointed at a scratch folder; the agent CLI we start reads
|
||
the same variable, lost its own provider settings, and failed with "unsupported provider". Back up
|
||
`config.toml` and restore it after the run instead, and leave the agent's environment as it is.
|
||
|
||
**A renamed CLI can't start the builder's tools.** To keep a live run's binary from being locked, it
|
||
was copied as `cli-r25.exe`; the CLI finds itself by name (`craft-anything-cli`) to start the
|
||
builder's MCP server, so Claude Code saw no tools and stopped. Copy it under its own name into a
|
||
folder of its own.
|
||
It happened again in a release: the macOS app bundle had only `craft-anything`, with the CLI in
|
||
a separate zip, so every Claude Code build from the 0.2.0 DMG failed the same way (#91). Every
|
||
package that ships the app ships the CLI beside its executable, and the packaging script runs
|
||
the CLI from the finished package before it's published.
|
||
Its symptom misled people too: Claude Code's MCP log says `Executable not found in $PATH: "stdio"`
|
||
for any command that won't start, so #64 went looking for a bad config. Reproduce with the real
|
||
program before believing what a log's wording suggests.
|
||
|
||
**A Windows path with forward slashes read as nothing.** rtl_433 decoded a recording in the shell
|
||
but found no messages, and exited 0, when the test passed `C:/…/x.cu8`; it only reads Windows
|
||
paths with backslashes. Hand external programs native paths (`\` on Windows), and give each
|
||
external tool a live test that checks it really produced output, not just that it exited 0.
|
||
It happened again with Explorer: project folders are kept as `D:/Projects/…\name`, and "Open
|
||
folder" opened Documents. Paths for the system's programs go through `platform::native_path`.
|
||
|
||
**A message on the command line.** Claude Code was handed the whole task as `-p <message>`; a
|
||
34,641-character brief passed Windows' 32,767-character command-line limit, so the build never
|
||
started ("The filename or extension is too long", os error 206, #402). Give a program anything
|
||
that can grow (messages, briefs, lists of files) on stdin or in a file, never in argv.
|
||
|
||
**Console windows popping up over the app.** Only the detached launcher hid the console; the
|
||
supervised runner didn't, so Ghidra's `.bat` and PowerShell each opened a console window during a
|
||
real Deep-mode run (the CLI never shows it: it already has a console). Every child started from
|
||
the windowed app gets `CREATE_NO_WINDOW`; check the desktop app, not only the CLI, after adding a
|
||
program.
|
||
|
||
**A graphics backend that crashed before it was chosen.** We ranked GL last when picking an
|
||
adapter, but eframe starts every backend first, and starting GL loaded an old Intel OpenGL driver
|
||
that crashed the app before any window or error box (#196). Ranking can't protect you from a
|
||
backend that's started at all: start only the backends the app may use.
|
||
It happened again with Vulkan on Windows (#394): starting Vulkan loaded an Intel Vulkan driver
|
||
(`igvk64.dll`) that crashed with `0xc0000005` on startup. Windows now starts DX12 only by default
|
||
(which includes WARP); `WGPU_BACKEND` still allows choosing other backends.
|
||
|
||
**A shader compiler picked by a default that looks around the computer.** wgpu's DX12 `Auto`
|
||
compiler takes a `dxcompiler.dll` it finds on the PATH; one without `dxil.dll` makes unsigned
|
||
shaders, and on an RTX 5060 Ti the device was lost as the window opened ("Parent device is
|
||
lost", #522). Where a library chooses a program or DLL by searching, name the one that ships
|
||
with the system (FXC), and keep the environment variable (`WGPU_DX12_COMPILER`) as the way to
|
||
another. A window that can't start now says how to draw without the graphics card
|
||
(`CRAFT_ANYTHING_RENDERER=software`). Not run on the reporter's machine or on Windows here.
|
||
|
||
**"Not installed" for a program that was there.** The system answers "not found" both for a missing
|
||
program and for a file whose interpreter is missing (an npm script with no `node`), and we turned
|
||
every one into "isn't installed or isn't on your PATH": #95, #130 and #152 reinstalled what they
|
||
had. When a spawn says not found, look the program up yourself; if it's there, name the file and
|
||
what it needs (`spawn_error_for`). Not run on Windows here.
|
||
|
||
**An answer with two parts, read as one.** An approval card has "Allow once" and "Always for this
|
||
project": the answer carries `allow` and `always`. The consent card our MCP server shows for a tool
|
||
that runs code read only `allow`, so "Always" was dropped and the same tool (a scripted web session)
|
||
was asked about every time (#563). When a card offers a choice, whoever reads the answer handles
|
||
every field of it, and a test answers it each way and asks again.
|
||
|
||
**A loop that only ends at its step cap.** A builder on an API backend may take 400 steps in one
|
||
step of the run, and each step sends the whole conversation again: one stage spent 9.4M tokens on
|
||
code that never compiled (#399). A failed build command is an `Ok` tool result (`succeeded: false`),
|
||
so counting tool errors misses it. Cap a loop on what it is stuck on (build commands failing in a
|
||
row, read from the typed `succeeded` field), not only on its total steps.
|
||
|
||
**A context overflow that looked like a broken request.** Ollama fits a prompt that is too long
|
||
for its context by dropping the oldest messages, keeping the system prompt; the person's request
|
||
went first, and a Qwen 3.8 model refused the rest with 500 "no user query found in messages"
|
||
(#1245). It read as a fault in our request, and as a server error it was retried three times. The
|
||
error came from Ollama's source (`server/prompt.go`, `model/renderers/qwen35.go`), not from the
|
||
words: when a local server's error makes no sense for the request sent, read what the server does
|
||
to it first, and mark a refusal that can't change on retry with a typed field (`context_overflow`).
|
||
|
||
**A limit left unset is the largest one.** We sent OpenRouter no `max_tokens`, so it held the
|
||
model's whole output limit (131,072 tokens for Kimi K3.1 Pro) against the key's credit, and a key
|
||
with $35 left of its daily limit was refused with 402 before the model ran (#1213). When a service
|
||
bills or reserves by a request's limits, find out what it assumes when one is missing; when it says
|
||
what it can afford, ask again within that.
|
||
|
||
**A router that picks the provider for you.** OpenRouter sends each request to one of several
|
||
upstream providers, and a provider without tool calling answered 400 "Provider returned error"
|
||
(#369), which said nothing the person could act on. For a service that routes, say in the request
|
||
what the provider must support (`provider.require_parameters` for tools), and show the provider's
|
||
own reason when it still fails (#542).
|
||
|
||
**A refusal that names a person nobody asks.** Auto mode refused `git push` with "never …
|
||
without the person", but never showed a card, so the person's "yes, push it" in the prompt box
|
||
reached only the builder, which retried and was refused again, five times. Its rule matched
|
||
`gh repo` anywhere, so `gh repo view` was "publishing" too, and "always" on `cd app && npm i` was
|
||
kept as `bash:cd`, which would have let `cd app && git push` through. A refusal says how the
|
||
person can allow it, and a way to allow it exists; read commands word by word; remember a
|
||
dangerous kind whole, never by a command's first word.
|
||
|
||
**A crash that no code can report.** Three Windows tickets (#33, #104, #74) said the app "does
|
||
nothing": a graphics driver crashed natively as the window opened, so no dialog, panic hook or log
|
||
line ever ran, and each fix only covered the driver already seen. When a failure can't be caught,
|
||
leave a mark before the risky step and clear it after (`start_guard.rs`); a mark found at the next
|
||
start is the report, and that start takes the safer path (WARP). Not run on Windows here.
|
||
|
||
**A spinner that kept the window drawing at full speed.** egui's `Spinner` asks to be redrawn at
|
||
once, every frame, while it's on screen. A build or a page capture can run for an hour, so the
|
||
window was redrawn at the display's rate the whole time (a core busy, or every core when drawing in
|
||
software), and a person reported "my CPU is fully used" (#579). Animations ask for their next frame
|
||
at their own pace (`widgets::animate`, `widgets::Spinner`); check a library widget for
|
||
`request_repaint` before using it for something long. Not measured on Windows here.
|
||
It happened again with egui's `ProgressBar::animate(true)` in the jobs list, which the spinner fix
|
||
didn't look at: replacing one widget isn't checking the others. Search the library's widgets for
|
||
`request_repaint` (`grep -rn request_repaint ~/.cargo/registry/src/*/egui-*/src/widgets`), and the
|
||
test that refuses egui's spinner on screens refuses its progress bar too.
|
||
|
||
**Every tool offered to a model that can only use a few.** The builder on an API backend was sent
|
||
all 33-41 Core and Build tools, about 23,000-30,000 characters of descriptions. A local 14B model
|
||
(#417) called the study tools (`core_session_show`, `core_journal_read`) 35 times and never
|
||
`build_file_write`, and Ollama's default context is smaller than the list. Offer a model the tools
|
||
the job needs (`LOCAL_BUILDER_CORE`), measure the list's size with the real registry, and run a
|
||
small local model before saying it works. Not run live here.
|
||
|
||
**A debugger failure reported as an observation.** `apple_calls_observe` built a Python callback
|
||
with `\n` between statements in a Rust string, so lldb rejected it and exited 1 before launching
|
||
anything; the tool ignored the exit status and returned "Observed 0 call(s)" with confidence
|
||
observed (#251). A run that never started the thing it observes is a failure, not a count of zero:
|
||
check the exit status before building evidence, and compile generated code (here, with Python)
|
||
before trusting it. lldb itself was not run here.
|
||
|
||
**A graphics default that drew on the other card.** egui-wgpu picks the discrete GPU unless told
|
||
otherwise; on a Linux laptop whose integrated GPU drives the panel, every frame was copied across
|
||
to it, the window took 20 s to open and the whole desktop lagged (#571). Forcing the low-power GPU
|
||
instead breaks desktops whose monitors are on the discrete card. Draw on the GPU that drives the
|
||
screen, read from the connected displays, and keep wgpu's environment variables as the override;
|
||
when a window is slow to open, check which adapter `desktop.log` says it drew with.
|
||
|
||
**A Metal version taken from the OS, not from the compiler.** wgpu-hal picks the Metal shading
|
||
language from the macOS version alone; on a 2012 iMac under OpenCore Legacy Patcher, macOS 15 asked
|
||
for `metal3.2`, the compiler stopped at 3.1, every shader failed and the window never opened (#742,
|
||
upstream #10687). Our `[patch]` of wgpu-hal probes the compiler (`vendor/wgpu-hal/CRAFT_ANYTHING.md`).
|
||
The advice on a lost device named `CRAFT_ANYTHING_RENDERER`, which only Windows reads: check that
|
||
advice names only settings the platform actually reads. Not run on the reporter's Mac.
|
||
|
||
**A Mac app that couldn't find Claude Code, and offered a GitHub key instead.** An app opened from
|
||
the Dock gets the system's short `PATH`, not the shell's, so `claude` in `~/.local/bin` wasn't
|
||
found; "Built by" then fell back to "Claude Code on github.com-work", an `~/.ssh/config` alias
|
||
for a second GitHub key, read as a computer to build on. Look programs up and start them with
|
||
`which::search_path` (`Process` and `Launch` do), and leave git servers out of SSH hosts.
|
||
|
||
**Found on Windows, but "isn't on your PATH" when run.** npm installs Gemini CLI, Codex and often
|
||
Claude Code as `gemini.cmd`-style shims; `which` finds them through `PATHEXT`, but Rust starts
|
||
only `.exe` files by bare name, so every build failed (#18). Through `cmd.exe` a prompt with line
|
||
breaks or over 8191 characters wouldn't arrive anyway. `Process` runs an npm shim's script with
|
||
Node directly (`which::launch_target`); test a new program on Windows with a multi-line argument.
|
||
npm also writes an extensionless POSIX script beside each shim, and `which` returned that first,
|
||
so OpenCode was offered and then every run said it wasn't installed (#41): look up the kinds
|
||
Windows can start (`PATHEXT`) before the bare name.
|
||
And when the shim's Node couldn't be found (nvm for Windows with no version in use, so its
|
||
`C:\Program Files\nodejs` link was gone), we fell back to the `.cmd`, and Rust refused the
|
||
multi-line prompt: "batch file arguments are invalid", which named neither Node nor nvm. Search
|
||
nvm's version folders too, and when a shim's interpreter is missing, say so (`interpreter_missing`).
|
||
The desktop DeepSeek Harness has a custom batch launcher, so npm-shim handling does not unwrap
|
||
it either. Send its task on stdin, with no task argument. A literal `-` is a stdin marker in newer
|
||
Harness versions but is the task itself in older ones; omitting the task makes those older versions
|
||
fail before a model runs. They need an update to a headless profile that supports stdin.
|
||
|
||
**A first analysis that waited for a lock it could never take.** Ghidra's analysis lock is created
|
||
with `create_new` in `ghidra-cache/`, which only existed on machines that had used it before; on a
|
||
fresh store the create failed and the loop read that as "someone else holds it" until the
|
||
deadline. Create a lock's folder first, and treat any error but "already exists" as a failure.
|
||
|
||
**A bound read from an analysis that never saw the code.** Ghidra's unguarded jump tables were
|
||
first ended at the next address something refers to; a hand-made ELF passed, but in a real Rust
|
||
program the code holding the next table's `lea` is reached only through the table being bounded,
|
||
so it was never disassembled, nothing referred to the next table and the bound took all 256
|
||
entries. When a fix leans on what an analysis found, ask whether the thing being fixed is why it
|
||
found nothing, and run it on a real program, not only on one built for the test.
|
||
|
||
**A bridge that didn't compile.** A helper in the Ghidra bridge was named like a public
|
||
`GhidraScript` method, so Ghidra refused the script and the run only said the bridge stopped.
|
||
Compile the `.java` against Ghidra's jars (`javac -proc:none -cp <Ghidra/**/lib/*.jar>`) before a
|
||
live run. Ghidra also rejects project paths with a folder starting with a dot, as
|
||
`tempfile::tempdir()`'s do: give live-test temp folders a plain prefix.
|
||
|
||
**A check that read another worktree.** A PR was tested in a worktree that shared the main
|
||
`target/`; `xtask` finds the repository through `CARGO_MANIFEST_DIR`, fixed when it's built, so
|
||
afterwards `cargo xtask` in the main tree reused that binary and read the worktree's files. Once the
|
||
worktree was removed it said "can't read docs/ROADMAP.md"; while it existed, every check that
|
||
reads files itself (`pending`, `secrets`, `layers`, `locks`, `tools --check`…) would have looked
|
||
at the wrong tree (the cargo commands `ci` runs use the folder they're started in). Give every
|
||
worktree its own `CARGO_TARGET_DIR`, and rebuild `xtask` (or delete it from `target/`) after
|
||
sharing one.
|
||
|
||
**A helper stopped only by a destructor.** The keep-awake helper was stopped when its guard was
|
||
dropped; closing the desktop app during a build ended the process without dropping it, and the
|
||
PowerShell loop kept the computer awake (its time limit was a week). A helper that outlives a
|
||
crash is a leak: have it watch the process it serves and end by itself (`caffeinate -w`, a loop
|
||
that checks the PID), and test by killing the owner, not by dropping the guard.
|
||
|
||
**Stopping a long run leaves children behind** unless every program goes through
|
||
`craft_anything_platform::process`. After stopping one, check that its child processes are gone.
|
||
|
||
**Separate target folders fill the disk.** Agents given their own `CARGO_TARGET_DIR` under
|
||
`target/` (to avoid locking a running exe) left them behind; one reached 79 GB and the disk filled
|
||
mid-build ("not enough space", os error 112). Delete an agent's target folder when its work is
|
||
merged, and check `du -sh target/*` when a build fails to write.
|
||
It happened again from a commit: the pre-commit hook builds `cargo xtask secrets`, and in a new
|
||
worktree with no `CARGO_TARGET_DIR` set that is a whole fresh build in its `target/`, which filled
|
||
the disk under a person's running build. Commit with the same `CARGO_TARGET_DIR` you built with,
|
||
and check free space (`df -h`) before a build next to someone else's work.
|
||
And again from the main `target/`: its incremental cache alone had reached 46 GB (81 GB in
|
||
`target/debug`) and the drive hit 0 bytes mid-test. Delete `target/debug/incremental` when it
|
||
grows, and build checks with `CARGO_INCREMENTAL=0`.
|
||
|
||
**A green PR that ran nothing.** `gh pr view` said `CLEAN`/`MERGEABLE`, but while the repository is
|
||
private every CI job is skipped, so nothing had been checked. Read the checks' conclusions, not the
|
||
merge state. Without CI, run its steps locally with `--locked`; a Mac can compile for the other
|
||
systems with `cargo check --target`, with zig as the C compiler for `ring` (a wrapper that drops the
|
||
`--target=` flag `cc` adds), but that only proves it compiles.
|
||
|
||
**A "required" switch read only on the failure path.** The build script looked at
|
||
`CRAFT_FONTS_REQUIRED` only when `CRAFT_FONTS_DIR` was set and bad; with the folder variable
|
||
missing, a build told the fonts were required finished with none embedded, and Korean showed as
|
||
boxes. A requirement is checked on every path that ends without the thing, the missing input first;
|
||
keep the decision in a function a test can call (`plan` in `crates/ui-egui/build.rs`).
|
||
|
||
**Unformatted code reached `main`.** A test was edited by script after `cargo fmt --all` had run,
|
||
so #53 merged with a long line rustfmt rewraps. Run `cargo fmt --all --check` last, after every
|
||
edit, right before committing.
|
||
|
||
**Clippy passed on an older Rust.** #1480 ran clippy on Rust 1.97 (the workspace's
|
||
`rust-version`), clean; CI's 1.99 has `clippy::chunks_exact_to_as_chunks` and stopped the PR on
|
||
it. A clippy that predates a lint can't report it. Run fmt and clippy with CI's toolchain
|
||
(`RUSTUP_TOOLCHAIN=1.99.0`), and if a tool says it isn't installed for a toolchain, install that
|
||
toolchain's component rather than running another copy.
|
||
|
||
**A process list read nothing on FreeBSD.** `ps -A -o pid=,ppid=` lists both columns on Linux,
|
||
but POSIX makes everything after `=` the header, so FreeBSD's `ps` printed one `pid` column headed
|
||
",ppid=" and every parent lookup came back empty. Give each column its own `-o` (`-o pid= -o ppid=`)
|
||
when a header is renamed or blanked.
|
||
|
||
**`cfg(unix)` tests assumed every Unix could capture a console program.** Two console-run tests
|
||
from #1679 and #1702 failed every night on FreeBSD, whose capture was a stub that refused to run
|
||
anything. A test gated on `unix` must only need what every Unix build has; FreeBSD now captures
|
||
through pipes, as Windows does.
|
||
|
||
## Design
|
||
|
||
**A source inventory limit came after expansion.** Taking five thousand names after collecting
|
||
every source path copied a large `sourceRoot` into every entry first. Count retained entries and
|
||
expanded UTF-8 bytes before allocating each path, share the budget across indexed sections and
|
||
application maps, and report when only a prefix could be kept. A source identity stays complete;
|
||
shortening one to fit the budget would describe a different source.
|
||
The source-to-bundle comparison then treated omitted paths as confirmed missing because it read
|
||
only the retained names. Carry inventory truncation through the diff and its evidence: preserve
|
||
positive matches, report the other sources as uncertain, and keep confirmed unmatched sources
|
||
for complete inventories. Test the consumer with a real exhausted budget, not only the flag.
|
||
|
||
**A recovery manifest repeated expanded source roots.** The source-map inventory budget did not
|
||
cover the recovery tool, which copied every expanded path into its manifest: a 75 KiB map with a
|
||
64 KiB root and 1,200 sources retained about 78 MiB. Apply the inventory budget to recovery paths
|
||
too, stop retaining originals when it is exhausted, and report partial recovery with a typed flag.
|
||
|
||
**An unread part of a source map looked like absent sources.** An index-map section can point at
|
||
another map (`url`), which the parser never fetches: it kept the section boundary but no count, so
|
||
a source comparison reported a source only that map lists as confirmed missing. Count unread
|
||
external sections on the map and in its inventory, carry the count into the application graph with
|
||
a limitation, and give an otherwise unmatched source an `uncertain` reason. Test an inline section
|
||
beside an external one, a nested index map, and the sources-only read used for bad mappings.
|
||
|
||
**A fix landed in one of two copies of a reader.** The `sourceMappingURL` rules were fixed in
|
||
`formats::sourcemap`, but `js::find_map_url` kept requiring a literal space, so a bundle with the
|
||
standard no-space annotation was analysed without its map. Have one implementation and call it
|
||
from both places; a regression at the consumer's public entry point catches a copy that drifts.
|
||
The same happened with the PE optional header: `pe::headers` was bounded by `SizeOfOptionalHeader`,
|
||
but `dotnet::pe::parse_pe` still read data directory 14 at a fixed offset, so a short declared
|
||
header could borrow the CLI directory from section-table bytes. When a structure's declared size
|
||
is enforced, search every reader of its fixed offsets and test each one with a short declared size.
|
||
|
||
**A carver read optional-header fields the header didn't declare.** The firmware scanner read the
|
||
PE certificate table and `SizeOfHeaders` at fixed offsets even when `SizeOfOptionalHeader` ended
|
||
first, so a short header could over-carve into the next embedded object. Bound every fixed-offset
|
||
field by the declared header, and test a short declared size beside the full-header control.
|
||
|
||
**A floating-point conversion changed a fingerprint.** Widening a single-precision NaN to a double
|
||
can change its sign, payload or signaling bit. Keep the original encoded bits for identities, and
|
||
test signaling/quiet pairs as well as different payloads; retain numeric conversion for display.
|
||
|
||
**A canonical-length check hid readable signature facts.** Rejecting a redundant ASN.1 length
|
||
made its certificate or signature disappear from the report. Keep a bounded, readable element,
|
||
mark its non-minimal encoding, and carry that limitation into every signature view. Test retained
|
||
metadata and the warning together; encoding conformance and cryptographic verification are separate facts.
|
||
|
||
**An unknown header bit hid readable code.** Rejecting undefined `.pyc` flag bits made the whole
|
||
file unreadable, dropping its code, names and constants from reports. A reader collecting evidence
|
||
must preserve fields and payload it can still understand: report the unknown bits as a warning,
|
||
derive known modes from their defined bits, and test that the warning and recovered data coexist.
|
||
|
||
**A round of checks that nothing could stop.** The builder's own checks ran one after another on
|
||
the build's thread with no time limit, Pause was looked at only between tool calls, and the feed
|
||
said only "checking its work against the original" for the whole round, so a check that stalled
|
||
named no check and could hold the build for hours. Give a builder's checks a round time limit,
|
||
look at Pause before each check, and say each check as its own step. The limit must reach the
|
||
checks' own program runs too: a build step (a game engine's import, a cold compile) makes no tool
|
||
call, so a limit that only applies "at the next tool call" leaves exactly the step that hangs.
|
||
A check that's cut off is a failed check with its reason, never a silent wait. And every way a
|
||
run can end — including a caught bug — records the project's status: leaving the last status on
|
||
disk ("Checking" while nothing runs) reads as a stuck build that can't be retried.
|
||
|
||
**An early error without a terminal event.** Model validation returned after reporting an error,
|
||
but headless chat waits for `TurnDone`, not just an idle worker. Preflight failures and cancellation
|
||
must finish the event sequence too; regression tests check both the error/status and the terminal event.
|
||
Worker-thread creation failures need the same terminal event even though no worker ever started.
|
||
|
||
**A conditional call erased the skipped path's value.** The ARM decoder discarded conditions on
|
||
calls, and value flow replaced the earlier register definitions as if every call executed. Keep
|
||
the call's condition and merge its possible writes with the skipped path in both block transfer
|
||
and edge construction. A known non-returning target stops only an unconditional call; after a
|
||
conditional one, only the skipped path contributes values. Test both kinds within one block and
|
||
across blocks, with unconditional calls as controls.
|
||
|
||
**A candidate cap discarded a real dependency.** The esbuild module reader kept the first 4096
|
||
resolved zero-argument calls, then filtered for known wrappers; ordinary local calls could fill
|
||
the buffer before a real dependency appeared. Discover wrapper bindings first and count only
|
||
recognised dependencies against their limit. Keep the buffer bounded, and test unrelated calls,
|
||
forward declarations and more actual dependencies than the limit permits.
|
||
|
||
**A capped window's counts read as totals.** A source-to-bundle comparison stopped examining after
|
||
`max_rows` scripts but put its matched/unmatched counts beside the total script count, and reported
|
||
the truncated unclaimed rows as the number of unclaimed units. Report how many were examined, count
|
||
totals separately from listed rows, and mark the cap in the result and the tool's limitation.
|
||
|
||
**A row cap kept the counts but lost the rows silently.** An archive comparison counted every
|
||
added, removed, changed and moved file, then listed only `max_rows` of each with no flag, so a
|
||
reader took the lists for complete. Set `rows_capped` beside the counts and report partial
|
||
coverage with a limitation, as the managed comparison already does.
|
||
|
||
**A report showed a prefix of its limitations as the whole list.** `core_report_write` rendered
|
||
only the first 30 distinct limitations in Markdown and HTML with no marker for the rest, so a
|
||
reader could mistake a capped list for everything the evidence said. Keep the display cap for
|
||
readability but name how many were omitted in both formats; test one more limitation than the cap.
|
||
|
||
**A comparison forgot what each side couldn't see.** `app_graphs_compare` and `js_exports_compare`
|
||
loaded application graphs but built their evidence with only their own limitations, so a malformed
|
||
source map or a truncated inventory disappeared and the comparison read as complete. Carry each
|
||
loaded side's limitations into the comparison evidence, labelled left/right, and test both path and
|
||
saved-evidence inputs.
|
||
**A reference list was cut and looked complete.** `native_strings_find` kept 32 references per
|
||
string and gave no count or flag, while the tool promised a complete search and the browse tool
|
||
already reported a full count. Keep the row cap for size but return `reference_count` (complete
|
||
within the analysis unless its xref limit was reached), a per-row `references_capped` flag, a
|
||
display-cap limitation and the analysis `xref_limit` limitation; test an engine-boundary program
|
||
whose string exceeds the display cap while the analysis limit is reached, including paging.
|
||
|
||
**A comparison list was taken for the whole set.** The application comparison cut added, removed,
|
||
bundle-module and vendor rows at 2,000, and its summary counted those cut lists as totals; the
|
||
export comparison cut each module's added/removed diff the same way. Keep a complete total beside
|
||
every capped list, aggregate `rows_capped` for the result, including nested function/export diffs of
|
||
changed modules, use the totals in summaries, and report partial coverage. Test more rows than the
|
||
cap on every capped path, not only the first one.
|
||
|
||
**A list cap stopped at one parsing path.** The syntax-tree pass bounded `endpoints` and `exposed`,
|
||
but the lexer fallback, used for huge, deeply nested or unparsable sources, pushed to both without
|
||
a limit, so the path chosen for hostile input was the one that could grow. Apply the same cap to
|
||
every push on every path, record the list in `truncated` where the parsed path does, and test the
|
||
fallback with more calls than the cap permits.
|
||
|
||
**A nesting guard missed the shapes it exists for.** The JavaScript nesting estimate counted
|
||
brackets and some keywords, but an `else if` chain lost its depth at every `;`, `extends` and a run
|
||
of prefix `++` counted nothing, and the tokenizer read `/` after `++`, after a member name or after
|
||
`yield` as a regex, hiding the brackets in the text after it. The `;` that ended each link lost the
|
||
chain depth when a comma expression came before it, a `;` inside a block consequent reset the chain
|
||
the block belonged to, and a regex after `yield` needs its brackets and prefix-operator runs counted
|
||
too, since the same text is division in sloppy code. Sources that estimated one or two levels then
|
||
overflowed the parser thread's stack and aborted the process. Count the `else` links at their own
|
||
scope, read a `yield /…/` as both a regex (for the fallback's facts) and possible code (for the
|
||
guard) with lookahead bounded across the source, ended by every line terminator (an escaped one
|
||
included), failing closed when its budget runs out, when a quote or backtick inside it starts a
|
||
string in sloppy code, when its closing slash would open a comment or a later token would differ
|
||
between the regex and division readings, and keeping the division reading's operator depth in the
|
||
expression so later nesting adds to it; test that a deep source reaches the lexer fallback with a
|
||
nesting reason instead of being parsed.
|
||
|
||
**An occurrence label was also a binding name.** The second `obj.x` window and the first
|
||
`obj["x[2]"]` window both appeared as `obj.x[2]`. Reserve literal labels and use a separate
|
||
disambiguator for collisions; never change the occurrence index to make a label unique. Include
|
||
unchanged windows when reserving labels so a different set of changes cannot rename a window.
|
||
|
||
**Reading literals and skipping the rest.** The Electron window reader skipped a spread inside
|
||
`webPreferences`, so a window whose preferences came partly from another object was reported as
|
||
"contextIsolation is not set; the default applies", which nobody had checked until a live run on
|
||
an installed app. When a reader takes literal values, anything that may add or replace them
|
||
(spreads, computed keys, later writes) makes the result unknown, never the default; and check new
|
||
readers on real, minified inputs, not only on the shapes the tests were written for.
|
||
|
||
**A naming scheme that clients can't accept.** Dotted tool ids (`web.page.capture`) were decided
|
||
before checking that MCP clients and model APIs allow only `[A-Za-z0-9_-]` in tool names. Check
|
||
external constraints (protocols, APIs, file systems) before fixing a public contract.
|
||
|
||
**A tool planned in a layer that can't reach what it needs.** Rebuild control as engine tools (L4)
|
||
can't call `RebuildRun` (L5). Before planning a tool, check which crate owns what it calls; if a
|
||
higher layer owns it, it needs a way for that layer to register tools (R1.7).
|
||
|
||
**An event queue drained in the middle of a run.** A scripted browser session's `goto` step
|
||
cleared the CDP event queue to "start fresh", so every network request before and during the
|
||
navigation vanished from the run's network capture. Only the code that owns a run's results
|
||
drains its event queue; steps leave events where they are, and the run keeps them all, tagged
|
||
with the step they arrived in.
|
||
|
||
**Exclusive creation left a permission window.** Opening a private temporary file with default
|
||
permissions could let another process retain a readable descriptor before a later chmod. Set
|
||
mode 0600 in the first Unix open for private files. Temporary names must also stay short and
|
||
independent of the destination name: adding a prefix and suffix can exceed the component limit
|
||
even when the destination name is valid. Keep the temporary file in the destination directory,
|
||
but exclude the destination component before creating a candidate, including case and trailing
|
||
dot/space aliases. Probe a supported long name in tests instead of assuming a fixed filename limit.
|
||
|
||
**Two writers to one file.** The app and the rebuild run both wrote `feed.json`. Give each persisted
|
||
file one owner; when another writer is unavoidable (an answer given from the CLI while a run is
|
||
going), merge what's on disk before saving instead of overwriting it.
|
||
The same happened in `craft.json`: a run saved from the copy it started with and undid a budget
|
||
the person raised meanwhile (and any "always allow" answer). Fields the person owns change only
|
||
through `Workspace::set_person`; `save` keeps them as they are on disk.
|
||
And again inside one process: the engine had its own `write_atomic` whose temp file was named
|
||
by process only, so two threads saving `craft.json` wrote the same temp file and one rename found
|
||
it gone (#273). There is one atomic writer, `craft_anything_platform::fs::write_atomic`; don't
|
||
write another.
|
||
The UI settings worker also released the session guard after cloning settings, letting that stale
|
||
snapshot overwrite a later model-selection save. Keep the guard through the worker's disk write
|
||
so session-based writers cannot overtake each other between reading and saving.
|
||
The exit-time save must merge the session's latest AI table before writing the app's pending settings;
|
||
the UI's AI snapshot may predate a selection already committed by the hub. But waiting for that guard
|
||
can hold shutdown behind a tool for hours. Try it once; if busy, durably defer only non-AI settings
|
||
and recover them at startup against the latest committed AI table, rather than racing a live writer.
|
||
After a successful full exit save, discard an older deferred snapshot so startup cannot replay it
|
||
over the newer app settings.
|
||
|
||
**Preflight checked a different transport.** An explicit OpenCode selection bypassed ACP, but
|
||
builder preflight still checked ACP's helper and the native adapter never supplied the MCP server.
|
||
Share transport selection between preflight and execution; reject unsupported native options
|
||
before running a process rather than starting an agent without the tools its task requires.
|
||
|
||
**A config route the installed version ignores.** OpenCode V2 (2.2.5) ignores
|
||
`OPENCODE_CONFIG_CONTENT`, and `opencode run` hands the turn to a background service that never sees the
|
||
run's environment, so the builder ran without our tools and fell back to calling the CLI from a shell.
|
||
V2 also names a tool call `id` (V1: `callID`), so every tool event was dropped and the feed stayed empty
|
||
for the whole turn. Use a file in `OPENCODE_CONFIG` with `run --standalone`, and confirm a live run
|
||
reached the server (`mcp connected server=craft-anything` in OpenCode's log), not just that it started.
|
||
|
||
**A side door to a service the main path avoided.** Runs used `--standalone` so OpenCode's shared
|
||
service never saw them, and #1185 deleted the sessions a stopped turn left. But the model check
|
||
before each turn ran plain `api model.list`, which starts that service, and the service resumed a
|
||
session the person had left unfinished days before and went on editing another project for half
|
||
an hour. When a program has a mode that must be avoided, give every call to it the same flag, not
|
||
only the main one: list each command we run (`run`, `api`, `session delete`) and test each.
|
||
|
||
**"Can it start here?" used for "is it a program?"** The study's search of a picked folder took
|
||
the biggest file `outputs::is_program` accepted, which on Linux and macOS means no extension and an
|
||
executable bit. A 1990s Windows game copied there has neither, so no program was found and
|
||
Deep mode skipped Ghidra; the tests named their programs the local way on each system, so none had
|
||
a Windows game on Linux. When a check answers one question, don't reuse it for a neighbouring one:
|
||
studying a program and starting it are separate (`studies_as_program`), and test the other
|
||
system's files on this one.
|
||
The fix itself first missed one gate that starts it: runtime study's console runs checked the PE
|
||
header but not the system, so a Windows console `.exe` would have been run on Linux (found in
|
||
review). When a value starts reaching code it couldn't before, check every place that acts on it;
|
||
every runtime start now asks `study::may_start`.
|
||
|
||
**Keywords that only some code paths honour.** "…in Rust" picked Rust for apps but not for games, so
|
||
a headless game rebuild stopped at once for want of Unity. It was found only by a live run. Run the
|
||
real flow end to end after changing how requests are read, and add the case as a test.
|
||
|
||
**A default given as an order.** "A notes app in Electron" was built in Rust: the brief quoted the
|
||
request, then said "Use Rust with eframe/egui… create a cargo project" with nothing saying which
|
||
wins, and the builder followed the later, more specific line; the app, the build check and Play
|
||
followed the stack picked at the start too. Say that the person's words come first and phrase a
|
||
default as one ("unless their request says otherwise"); let what acts on the result (checks,
|
||
Play, the recorded stack) look at what was built, not at what was planned.
|
||
|
||
**More than one way to build the engine.** The CLI's `commands` listing built its own engine
|
||
from default settings, so it missed the plugins (and the person's settings) every other command
|
||
had. Build the engine through one function per app (`make_engine` in the CLI) and add new layers'
|
||
tools and plugins there.
|
||
|
||
**A best-effort step that hides its errors.** Capturing a site's extra pages used `.is_ok()`, so a
|
||
missing parent folder made every page silently drop out; only a live run showed "Photographing more
|
||
pages" followed by nothing. When a step may fail without stopping the run, still say why (the feed,
|
||
a note, a log line), and test the success path end to end, not just the picking.
|
||
The fix then made the check create `checks/round-N/routes/` first, which made `round-N` exist, and
|
||
`web_site_capture` (which refuses an existing folder) stopped the run. Create a tool's output parent
|
||
right before that tool runs, never a folder that another tool must create itself.
|
||
|
||
**A size measured after drawing, used a frame late.** The prompt box's height was measured each
|
||
frame and used to size the feed on the next, so every time it grew or shrank one frame showed it
|
||
100 px too low, and typing looked like flicker. When a layout uses a size from the last frame,
|
||
call `ctx.request_discard` when it changed, so egui lays the frame out again before showing it.
|
||
The same layout then came out 17 points taller than the space it was given: it left out the item
|
||
spacing after the feed, and `ui.horizontal` centred the 85-point box 12 points down in its row.
|
||
The agent console's panel grows to fit its content, so it grew 17 points every frame until its
|
||
maximum, and then the top of its tabs was cut off. When a part is sized to fill what's left, count
|
||
the item spacing and use `horizontal_top` for tall rows, and test that the panel's size holds still
|
||
over several frames.
|
||
Its first frame had no measure yet and guessed 48 points for a box of 84 or more, so the panel grew
|
||
on that frame, and egui saved the larger size (#942). Where a panel grows to fit, guess too tall.
|
||
|
||
**A library default that blinked over a text field.** egui's scroll areas fade their edge when
|
||
there's more below; the prompt box's one measured a few points of overflow on the frame after each
|
||
keystroke, so a dark gradient blinked over the line being typed. Unit tests look at text, not at
|
||
what's painted over it. When something flickers, render the frames (`Harness::render` at a
|
||
fractional scale) and diff them, then list the shapes painted over the spot.
|
||
|
||
**A tool that locked the session it was called with.** `rebuild_question_ask` read the settings
|
||
with `hub.engine.session()`, but a tool runs while its call holds the session, so the CLI waited
|
||
forever before starting anything (found by the first live run, not by tests, which stopped
|
||
earlier with no AI set up). Inside a tool, read what you need from `ctx.session` and hand it on;
|
||
test a tool through `Engine::call` with a time limit on a thread.
|
||
|
||
**A forked session's cost is the session's.** Claude Code's `total_cost_usd` for a
|
||
`--fork-session` run is the whole session's running total ($22.67 for a question that cost cents).
|
||
Don't show a cost you can't attribute to the thing it's shown beside.
|
||
The same holds for `--resume`: a build turn's total is the session's running total, so summing
|
||
each turn's total counted the session again every turn (#1187). Keep each session's last total and
|
||
add only the increase; a per-turn `usage` block is per turn, the cost fields are not.
|
||
|
||
**An option that offered nothing new.** Asked for "a shallow live copy", the pin dialog got a
|
||
Live choice that copied nothing and opened the project's app, which is exactly Play, under a
|
||
name that suggested something was kept. It was merged, tested and installed before anyone asked
|
||
what it was for. Before adding a choice, write one sentence on what it gives that the existing
|
||
path doesn't; if there's no such sentence, ask what was meant.
|
||
Its replacement nearly repeated it: hard-linking dependencies saved almost nothing on real
|
||
projects, because 85% of a Rust pin was a `.pdb` nothing needs. Measure on real data before
|
||
designing around where you assume the size is.
|
||
|
||
**Long content above a fixed prompt box.** Answers to questions were cards between the feed and
|
||
the prompt box, sized to their text; the feed got what was left, so a long answer squeezed it and
|
||
pushed the box off the window. Content of any length belongs inside the scroll area. And an X on
|
||
something that can't be brought back is a delete: offer folding instead, and keep the data.
|
||
|
||
**A library default that hid the scroll bars.** egui's thin scroll bars paint the thumb in the
|
||
button fill and the track in the field colour; our theme made those the same, and the thin
|
||
preset narrows the bar to 2 px at rest, so the thumb showed only on hover. When a theme sets a
|
||
library's colours, check every widget that borrows them, in both themes, and test contrast with
|
||
numbers (3:1 for parts you interact with), not by eye.
|
||
|
||
**Files read while drawing.** The Project screen parsed `package.json` 2–4 times a frame, listed
|
||
the reference pictures and checked screenshot files on every repaint, and read the side panel's
|
||
files on every frame a build event arrived; on a slow or network drive the window would freeze.
|
||
Drawing reads only memory: anything from disk is read on a worker when something changes, and
|
||
the screen shows the last result.
|
||
|
||
**A non-blocking read that falls back to defaults.** The app read the person's settings with
|
||
`try_session().unwrap_or_default()` at startup, so a busy engine meant Clean room and no AI, and
|
||
the next settings change saved those defaults over theirs. Seen once in a live run. A fallback
|
||
must never stand in for data that gets saved back: wait (it's startup), or keep what's on disk.
|
||
The model picker later captured a busy session as "no project", queried in the wrong directory,
|
||
and rejected the save as a settings change. Capture settings and project under one guard; report
|
||
contention, preserve the draft and matching background results, and check again when available.
|
||
It happened again in the loader itself: `Settings::load_from` took every read error for "no file",
|
||
so a config.toml that wasn't UTF-8, or that another program held for a moment, loaded as defaults
|
||
and the next save replaced it; a file that couldn't be parsed was renamed aside with the rename's
|
||
result ignored. Only "not found" means there's nothing to lose. Defaults that stand in for a file
|
||
that is there carry that fact (`Settings::not_loaded`) and refuse to be saved, also through a
|
||
serde round trip that would drop it (`with_setting`); a backup keeps every byte under a name
|
||
nothing else has, and one that fails stops the save.
|
||
And "not found" wasn't that either: reading a link whose target is missing answers "not found"
|
||
too, so such a config.toml was a first start and the next save put a file where the link was. A
|
||
read's error describes the end of the path it followed, not the path: before treating "not
|
||
found" as "nothing to lose", look at the path itself without following it (`symlink_metadata`).
|
||
|
||
**A save that changed memory first.** Adding an AI in the app put it in the session's settings,
|
||
then the file write failed (on Windows, likely a virus scan holding config.toml), so the app showed
|
||
the AI and its role while nothing was saved, and the error sat in the message bar behind the Settings
|
||
window. Save to disk first and change memory only after it worked; replace files with a short retry
|
||
on Windows (`write_atomic`); show errors where the person is looking.
|
||
|
||
**A limit on what is kept, taken for a limit on the work.** `walk_dir` stopped keeping files at
|
||
`max_files` but went on through the whole tree, read each folder's entries into one list before
|
||
looking at any limit, and kept every folder and link it met: a tree of empty folders or links, or
|
||
one folder with millions of names, had no limit at all. Count what is examined, not only what is
|
||
kept; stop at the first thing over a limit; bound what is held while collecting; and test the
|
||
shapes that never reach the limit on the thing counted (no files, only links, one huge folder).
|
||
The fix repeated it one level down: to keep a folder's first names in order it held only as many
|
||
names as the limit, but read the folder to its end to find them, 100,000 entries for a limit of 2,
|
||
and its test checked what was held, not what was read. A limit on the work is a count of reads:
|
||
take what the limit allows and one more (which says there is more), and assert how many times the
|
||
source was read with a counting iterator. That costs the stable order: a cut listing holds what
|
||
the system returned first, and the docs must say so rather than promise the same cut every time.
|
||
The same loop skipped entries that failed to read and returned what was left as the whole folder;
|
||
64 failures in a row came back as an empty folder listed in full, and a test expected exactly
|
||
that. Anything skipped while reading is reported with the result (a count, a reason, a typed
|
||
"incomplete" flag), and a test of an error path asserts what the caller is told, not only that
|
||
the loop ended.
|
||
|
||
**A result with two "what was missed" fields, read as one.** A folder walk says a listing is
|
||
incomplete in `truncated` and names the folders it couldn't list in `unlisted`; five tools read
|
||
only the flag, so an unreadable folder was "a limit" with no name and no reason, and a folder that
|
||
couldn't be listed at all set `unlisted` without the flag. When a result carries what was missed
|
||
in more than one field, check every caller reads all of them, and test each caller with the real
|
||
failure (a folder held without sharing, or without permission), not only the walk.
|
||
|
||
**An ignore file was read whole.** `walk_dir` read each `.gitignore` with `read_to_string` and
|
||
swallowed every error: a sparse oversized file could size the allocation until the process aborted,
|
||
a FIFO at that path could block the walk, and an unreadable or non-UTF-8 file lost its rules with no
|
||
limitation. Read it through the capped, regular-file-checked helper, keep the cap well above real
|
||
ignore files, and name every skipped file in the evidence; an absent file is the only silent case.
|
||
|
||
**An inventory's size used as a read limit.** The managed reconstruction check compared a file's
|
||
inventoried size with its 64 MiB budget, then read the file whole; growth in between bypassed the
|
||
budget. Use the capped regular-file reader with the remaining budget, which checks bytes read
|
||
as well as metadata. Test growth after a real inventory, the aggregate limit and exact-limit
|
||
inputs. When nothing can be read, keep the read failures and the walk cut in the error too.
|
||
|
||
**A graph cap was silent for edges and complete in coverage.** The application graph dropped
|
||
edges past `4 × max_nodes` with no counter, and `js_app_map` reported Complete even when nodes or
|
||
edges had been cut. Count every unique dropped edge, name the cap in a limitation, and make
|
||
coverage partial for either cap. Test a dense graph and a node-capped one.
|
||
|
||
**A fact list was cut and the graph looked complete.** `js_app` resolved only the first 2,000
|
||
imports per module and discarded `ModuleFacts.truncated`, so later dependencies lost their edge and
|
||
role with no limitation. Resolve every retained import, aggregate per-file truncation into one
|
||
named limitation, and make the map partial; test both the old 2,000 boundary and the facts limit.
|
||
|
||
**Showing settings shows their secrets.** `settings_show` returned the whole settings, plugins'
|
||
`env` values included, which can hold API keys; a tool's output reaches the journal and MCP
|
||
clients. The journal only redacts by field name. When a setting can hold a secret, hide its value
|
||
in every tool that prints settings, and test that it doesn't appear.
|
||
|
||
**Network access that became account access.** Restricted MCP hid the sign-in tools, but
|
||
`web_site_open`, allowed by `--allow-web`, picked a saved sign-in by default, so a client that
|
||
couldn't sign in could still browse the person's account. When a tool gains a default that reaches
|
||
stored credentials, check every mode that admits it, and give the narrower mode its own rule.
|
||
The same profile copy also carried the browser's saved-password store: "we never store a
|
||
password" has to cover what the browser itself offers to keep.
|
||
|
||
**A secret check that matched a word inside a name.** `cargo xtask secrets` took any variable
|
||
with `AUTH` in its name as a secret; git sets `GIT_AUTHOR_NAME` while the hook runs, so a contributor
|
||
whose git name is their handle couldn't commit their own roadmap claim. Name the exceptions
|
||
(`NOT_SECRET`) rather than loosening the word, and test both sides with a list of variables.
|
||
|
||
**Secret files kept out only by one person's gitignore.** `.env` files and
|
||
`.claude/settings.local.json` were ignored by a contributor's global gitignore, not the repo's, so
|
||
anyone else could have committed them. Ignore secret and local files in the repo's `.gitignore`,
|
||
and run `cargo xtask secrets` before every commit (the `.githooks/pre-commit` hook does it).
|
||
|
||
**An empty float sum is `-0.0`.** `iter().map(..).sum::<f64>()` over nothing gives `-0.0`, which
|
||
`{:.2}` shows as "$-0.00". Sum money with `fold(0.0, ..)`.
|
||
|
||
**A per-run budget that a retry resets.** The build's refinement cap (`max_rounds`) counted from zero on every
|
||
run, while the round numbers carried over, so "Try again" bought six more checks each press and a build that never
|
||
worked reached check 40 and 49 (#736). A limit that must hold across runs is a count saved with the project, and
|
||
it's reset only by a new request. Test it with several runs in a row, not one.
|
||
|
||
**A cap meant for a default, applied to what the person set.** The 150k limit that keeps a paid
|
||
service's Auto budget cheap was applied to the window read from the server before the person's
|
||
own budget was compared with it, so a 390k budget on a service with a 1M window became 150k
|
||
(found in review of #1155). Apply a default's limit only on the path where no value was set, and
|
||
test a set value above that limit.
|
||
|
||
**A question that started what it asked about.** The window probe skipped llama-swap's upstream
|
||
route for an unloaded model, but went on to llama.cpp's `/props?model=…` on the same server, and
|
||
llama-swap starts a model for that too: asking about a model with no window in the list unloaded
|
||
the one in use and loaded the other (seen live; the test had only checked that the upstream route
|
||
wasn't asked). Once a server is known to start models on request, ask it nothing but its list
|
||
about one that isn't loaded, and test the whole list of URLs asked, not one of them.
|
||
|
||
**One field, two meanings.** `TurnDone`'s `input_tokens` included cached tokens from every coding agent but not
|
||
from the built-in loop, which keeps them apart as Anthropic reports them. Once Ollama 0.40 reported its cache, a
|
||
local build's `usage.json` said 5,323 input tokens for 75,014. When several producers fill one field, say on the
|
||
field what it counts, and test each producer with the case where they differ (here, a cached prompt).
|
||
|
||
**Docs that drift.** `STATUS.md` described a branch 25 commits old and the tool count was stale.
|
||
Update STATUS, the roadmap and generated docs (`cargo xtask tools`, `cargo xtask schemas`) in the
|
||
same PR as the change.
|
||
|
||
**A tool's title is a UI string.** A PR that added an engine tool and no UI passed every check run
|
||
for it locally, and CI's `cargo xtask i18n-coverage` failed: the app shows tool titles, so each
|
||
needs a translation in every catalog. Run `cargo xtask i18n-coverage` for any PR that adds a tool
|
||
or changes a title, not only for UI changes.
|
||
|
||
**An inserted name became another placeholder.** Filling `{saved}` and then `{used}` with repeated
|
||
replacements rewrites a saved builder named `{used}`. Fill every placeholder in one pass over the
|
||
template (`platform::text::fill`, which `i18n::trf` uses), never by chained `replace` calls, so
|
||
inserted names stay literal; test placeholder-looking names as well as ordinary ones.
|
||
|
||
**A second lock in one statement.** `CliSpec { deep: !engine.session().clean_room(), logs: engine.session().config_dir(), … }`
|
||
looked like two quick reads, but an unbound guard lives to the end of the statement, so the second
|
||
waited on the first forever. Every build froze after studying its target while the card calmly
|
||
said "This is a big one", and no test ran that line. Read the session once; `cargo xtask locks`
|
||
and `platform::sync` now catch this. A wait message must not reassure about what it can't see:
|
||
say how long it's been since anything happened. Find a hang with `sample <pid>`: it names the lock.
|
||
|
||
**An error that points at nothing.** "Check the output below; run Claude Code by hand" came with no
|
||
output below (the event carried only message and remediation) and no command, and the program
|
||
that failed was really ssh, talking to a git alias from `~/.ssh/config` that had been offered as a
|
||
computer. Say what ran and what it printed, write the rest to a log the person can open (a failed
|
||
agent run does: `crate::failure_log`), and never name a next step you don't spell out. Exit 255
|
||
from ssh is ssh's own failure: don't blame the remote program. Test a message by causing the failure.
|
||
|
||
**A rule in the brief taken for a guard.** Every brief says everything outside `app/` is read-only,
|
||
and an OpenCode builder still wrote its own progress notes over `inbox/build-progress.json`. The
|
||
run read the record strictly, as it should, and every Try again stopped at "unknown field `step`"
|
||
with no way forward. A coding agent's own tools don't pass through our checks: a file the run
|
||
depends on is read as if anything may have changed it, and the error for one that changed says
|
||
what the file is and how to go on.
|
||
The fence in OpenCode's own rules was first drafted as "deny every edit, allow `app/*`", from the
|
||
brief's words; a project edited in place (`app_in_root`) has its app in the project folder itself,
|
||
and every edit its builder made would have been refused. A rule drawn from one layout's words
|
||
misses the other: build it from the typed list the code already keeps (`OWN_NAMES`), and test
|
||
both layouts.
|
||
|
||
**A retry that took "no message" for "brand new".** A run with no message and no `continuing`
|
||
flag was read as a first build, and Try again starts exactly that run. A browser that died during
|
||
the final check failed a finished build; Try again then studied the original again and gave the
|
||
builder the first brief. A run decides what's left from what the project records, not from which
|
||
button started it, and a check that couldn't look says nothing about the build.
|
||
|
||
**A stopped tool call broke the next request.** Retaining a stopped assistant reply kept its
|
||
tool calls without matching results. The next model request then carried unpaired calls.
|
||
Every retained call needs a matching result, including an explicit non-execution result.
|
||
Test the next serialized request as well as whether a tool ran.
|
||
|
||
**A deferred request reset lost on failure.** Moving consumed-note bookkeeping out of the live
|
||
builder loop saved the reset only after a successful turn. A failed or cancelled turn had already
|
||
removed the note, so reopening kept the old unchanged-turn count. Persist deferred request state
|
||
on terminal errors too, outside provider retries; keep the original outcome if that save also fails.
|
||
|
||
**Context added to the message but not the note.** "Just ask" answers were prepended to the
|
||
builder's next message, but a note can reach a builder through its tool hook before any message,
|
||
so the note carried the request without what it referred to. Put shared context where every path
|
||
reads it (the note's hand-over file and the after-turn prompt), not in one delivery path, and test
|
||
the fast path (hook) and the fallback (after the turn) separately. Mark it told only when the note
|
||
is actually read or sent, never when it is written; and keep a note's raw text so a later run can
|
||
filter answers by its own mode.
|
||
|
||
**Lower-casing a command, then comparing its paths.** Auto mode lower-cased a shell command to
|
||
match keywords, then checked its paths against the project folder with that same text; Windows
|
||
compares paths without case, so only a Mac showed it: deleting `/Users/…/app/Build` in the
|
||
person's own project was refused. Match keywords on a lower-cased copy and compare paths as
|
||
written; test a path rule with capitals and a folder outside the temp directory.
|
||
The same check split the command at every space, quotes included, so `"…/MyGame Backup/x"`
|
||
began with the project's path and went through; it had been noted as "only refuses too much"
|
||
without trying a folder named like the project. Read arguments as the shell does, and test a
|
||
path rule with the folder beside the target, not only the target.
|
||
|
||
**A filter written for one kind of text, run on another.** The stand-in's secret scan was written for
|
||
API answers, then also ran on recorded scripts: `this.token=t` became `[dropped]` and the scripts
|
||
broke, and "Basic Membership" lost a word. It also tested whole tokens, so a secret inside a URL or
|
||
after `ref=` passed. Test a scrubber on what must survive as well as what must go, and on each
|
||
kind of content it can receive.
|
||
|
||
**Response metadata that fills a replay limit.** Comparing stored HTTP headers literally kept
|
||
equivalent polls apart: names ignore case, and only values for the same name retain order.
|
||
Compare that way without rewriting the saved headers. Unavailable 304 samples still prove which
|
||
request lacked a cached body. At capacity, replace one only with a usable response for the same
|
||
path, query and request key, then append it in recording order. Keep other requests' evidence,
|
||
count the displaced sample, and do not replace anything with a body refused by a size or secret
|
||
check. Compare complete samples before charging retained bytes, so repeats need no refund. Test an
|
||
exact total-body limit and one with room remaining, preserving the unavailable 304 and remaining budget.
|
||
|
||
**A strict tool that sent an AI off to write its own.** `game_pak_extract` refused an output
|
||
whose folder didn't exist and took one file per call; the extractor model hit that twice, then
|
||
wrote its own pak reader in Python for a large archive. Tools an AI drives make what they
|
||
can (folders), take whole sets in one call, and every brief says to fix the call rather than
|
||
replace the tool. Read an AI run's transcript for tools it gave up on.
|
||
|
||
**Two engines, two address spaces.** Ghidra loads a position-independent ELF (lowest segment at
|
||
0) at 0x100000, while the built-in engine reads it where its headers say. Handing Ghidra our
|
||
address found nothing there, and Deep mode quietly fell back to the built-in decompiler for every
|
||
PIE program and shared library. Whenever an address crosses between engines, move it by their
|
||
image bases (`ghidra::Rebase`, from the base the bridge reports) and say in the evidence which
|
||
space each address is in.
|
||
|
||
**An id that only means something on one computer, kept in a folder that travels.** The project
|
||
saved Claude Code's session id, but the session itself lives in `~/.claude/projects/` on the
|
||
computer (and under the folder) that started it; a project synced to another computer stopped at
|
||
"No conversation found" (#452), and the retry for a cut-off turn asked for the same missing session
|
||
three more times. Anything a project keeps must still work where the folder goes: keep what the id
|
||
points to as well (`agent-session.jsonl`), tell "not here" apart from a failure by a typed signal
|
||
(`Conversation::session_lost`), and never retry what can't succeed.
|
||
|
||
**An installer that downloaded what was already there.** Ghidra was found only on PATH and at two
|
||
Linux folders, so an unzipped copy in Downloads, Program Files or a Homebrew cask was missed, and
|
||
the install then downloaded the same release again (#740). Install folders are a search, not a
|
||
list: scan the folders people keep such things in (`KnownRoot`, by folder name, a set number of
|
||
levels down), and make an install use a copy already found outside our store. Test each layout
|
||
(unzipped, cask, capitalised name) with temp folders, not the machine the test runs on.
|
||
|
||
**A per-run id that a provider expects to be stable.** OpenCode Go refuses a request without
|
||
`x-opencode-session` (#1225). The header was sent, but only for backends saved with an id, so
|
||
backends from older versions or found from a key got a new id on every run. A fallback id for one
|
||
run is a workaround, not a fix: give each saved provider backend its id once, at startup, before
|
||
any session lock is held (`AiHub::ensure_session_id`), and test a restart keeps the same id.
|
||
|
||
## Agent tooling
|
||
|
||
**Shared CLI efforts reached Claude unchanged.** Keep CLI-only levels available for agents, but
|
||
normalize at the provider boundary; test serialized requests for known and unknown model families.
|
||
|
||
**A sender could exceed its MCP receiver's target limit.** Bound the handoff to the receiver's
|
||
accepted count and retain the active target; test the boundary, truncation order and remote mode.
|
||
|
||
**A present protocol field was not a valid one.** The MCP server echoed `_meta.progressToken`
|
||
whenever it was present, including `null` and objects, so it sent progress for values the protocol
|
||
doesn't allow. Check a protocol field's type, not only its presence, before acting on it; test
|
||
missing, null, wrong-type and valid values.
|
||
|
||
**A finding correction reset omitted confidence to inferred.** Edit defaults must preserve existing
|
||
values unless explicitly changed; test omitted fields and every supplied confidence level.
|
||
|
||
**Atomic replacements still lost concurrent investigation updates.** A process-local mutex and
|
||
rename protect neither a cross-process revision check nor the whole read/modify/write sequence.
|
||
Hold one OS file lock from load through every rendered write, use it for all writers, bound and
|
||
cancel the wait, and test with separate processes. Never treat malformed state as an empty update.
|
||
|
||
**An optional view refresh could fail the builder's turn.** Log cache-refresh errors and continue
|
||
relaying agent events; only failures of the actual build belong on its fatal path.
|
||
|
||
**Generic ACP errors named Copilot for every adapter.** Use the current agent's display name and
|
||
the adapter's sign-in guidance; test an ACP-only adapter with and without that guidance.
|
||
|
||
**An agent's findings existed only in its own session.** A separate MCP process started without
|
||
the desktop's targets or a saved project, so its `fnd_*` records never reached the app. Local
|
||
agent chats now attach a shared investigation, hand over target metadata and refresh snapshots
|
||
after tool results. Test the child process's targets and the UI's finding ids, not only tool calls.
|
||
|
||
**A headless egui check reached rendering and then failed in teardown.** The app's named fonts
|
||
must be installed, and a test without a renderer must explicitly clear `output.textures_delta`.
|
||
Use the repository's `run_ui` API and its font setup instead of assuming an older egui harness.
|
||
|
||
**An ACP handshake passed while the tools were missing, and an empty tool list enabled them all.**
|
||
Copilot CLI 1.0.105 initialized successfully but did not expose a server supplied only through
|
||
ACP's `session/new`; explicit `--additional-mcp-config` made it available. An empty
|
||
`--available-tools=` restored native tools instead of disabling them. Verify the tools actually
|
||
sent to a fake local model, including the no-tools case, before trusting startup or argument tests.
|
||
|
||
**Shell quoting breaks generated edits.** Heredocs carrying Python with quotes, `'''` or `\n` were
|
||
mangled by the shell several times. Write multi-line edit scripts to a file and run the file.
|
||
|
||
**A Python edit that turned every line into CRLF.** `Path.write_text` on Windows writes CRLF
|
||
line endings, so a scripted size change rewrote 19 LF files with CRLF (git only warned). Write
|
||
with `write_bytes(s.encode())` and check `git ls-files --eol -m` after scripted edits.
|
||
|
||
**A process search that found itself.** Looking for a helper by a word in its command line, with
|
||
that word in the search's own command, listed the search: "a helper is left behind" after every
|
||
run, with a new pid each time. The new pid was the clue. Build the pattern from parts
|
||
(`'Set' + 'Thread'`), and don't stop processes by pattern on a machine where real builds may run.
|
||
|
||
**A command that waits for a terminal.** `git stash show -p` opened a pager and an empty
|
||
`python - <<'EOF'` heredoc opened the interactive interpreter; both sat waiting for input until
|
||
the tool timed out, with work left in a stash. Never run anything that can page or prompt: pass
|
||
`--no-pager` (or `GIT_PAGER=cat`), never feed an interpreter an empty script, and check
|
||
`git stash list` and `git status` after a timeout before carrying on.
|
||
It happened again from a leftover first line (`python -I - <<'X'` with nothing before `X`) in a
|
||
command that then wrote and ran a script: 10 minutes lost. Write the script with the editor
|
||
tool and run only `python -I <file>`.
|
||
And twice more in one night, each time a `python -I - <<'X'` with an empty body typed as a
|
||
throwaway step in a longer command. There is no safe use of `python -` here: never type it.
|
||
And again as the first step of a `sed` edit (`python -I - 2>/dev/null; sed …`): the command sat
|
||
for two minutes and the edit after it never ran. Make an edit with the editor tool, not a shell
|
||
line. The same session typed it once more, with `</dev/null` added, and that hung too: the habit
|
||
is the problem, not the input. Every Python step names a script file.
|
||
|
||
**A shell script tried in bash, run by dash.** A watcher script written for `/bin/sh` used
|
||
`kill -0 -- -<group>` and passed in Git Bash; on Ubuntu `/bin/sh` is dash, whose `kill` takes `--`
|
||
as a process id, so the script quit at once and both of its tests failed in CI. Try a `/bin/sh`
|
||
script with `dash` (or `checkbashisms`) as well as bash, and keep to what both accept.
|
||
|
||
**A search that ran no search.** `timeout 40 find … 2>/dev/null` on a Mac: there's no `timeout` there,
|
||
the error went to `/dev/null`, and the empty result read as "no such files on this computer". Don't
|
||
hide a command's errors when an empty result means something; set the time limit on the tool call.
|
||
|
||
**A file list in a variable that zsh didn't split.** `F=$(grep -l …); sed -i '' … $F` handed sed all the
|
||
names as one path; it failed, the tests that followed passed on the old code, and it read as done.
|
||
Pipe lists into `xargs`, and check the edit landed (`git status`) before reading the test result.
|
||
|
||
**A scripted edit that stops half-way.** One script deleted a file, then failed an assertion on a
|
||
later file, leaving the tree half-edited. Make edit scripts check every anchor before writing
|
||
anything, and run `git status` after any failure before trying again.
|
||
|
||
**A scripted edit that deletes more than it meant to.** A roadmap update found the end of an item
|
||
by searching for the next `- [` line; the item was the last in its section, so the search ran into
|
||
the next section and deleted its heading. Read `git diff` of every scripted edit before committing,
|
||
and bound searches to the section (stop at the next heading).
|
||
|
||
**A time limit that also covers the build.** A live browser test was run under `timeout`; the
|
||
first build of the test binary used most of the time, the limit killed the test mid-run and its
|
||
private browser was left running. Build first (`cargo test --no-run`), then run the test under the
|
||
limit, and don't kill test processes by hand to clean up after it.
|
||
|
||
**A UI tree that showed a password.** The Windows UI tree skipped values of password fields, but
|
||
.NET's UI Automation client hadn't loaded its Win32 proxies, so every WinForms control was a plain
|
||
pane, nothing was flagged as a password, and a password box's text came back as its title. Register
|
||
the client-side providers, never read a password field's title or value, treat an unrecognised edit
|
||
box as one, and test against a real window with a secret in it.
|
||
|
||
**A list of tool names that drifted.** The MCP server marked results as untrusted by a list of
|
||
tool names; two of its names no longer existed, so nothing failed and the gap went unseen. Put
|
||
such facts on the tool itself (`Effects::CARRIES_TARGET_CONTENT`), compare `Registry::canonical`
|
||
ids (which cover old ids and `mcp__…__` names) instead of raw names, and keep tool ids that other
|
||
code depends on in one constant next to the tool.
|
||
|
||
**A new tool named the old way.** Two new tools got ids outside the scheme although new tools use
|
||
`<area>_<object>_<action>` from the start, and had to be renamed. Read docs/tool-ids.md before
|
||
naming a tool.
|
||
|
||
**A stash that left a new file behind.** Work in progress was stashed to fix something on another
|
||
branch; `git stash` keeps only tracked files, so a new file stayed in the tree and `git add -A`
|
||
put it into the other PR. Use `git stash -u` (or commit the work in progress on its own branch),
|
||
and read `git status` before `git add -A`.
|
||
|
||
**Fence checks that disagreed about tool calls.** Suffix checks accepted malformed closing text,
|
||
while requiring exactly three closing backticks rejected valid longer fences. Share one boundary
|
||
parser between JSON recovery and ReAct: count the opener, require a standalone closing line at
|
||
least that long, and reject an earlier matching close followed by more text. Shorter backtick
|
||
lines inside a longer fence are content. Test both paths and the next HTTP request, including
|
||
malformed suffixes and non-call JSON examples whose strings contain protocol markers.
|
||
|
||
**A call the server couldn't read, taken for the answer.** A llama.cpp server turns a Qwen
|
||
model's `<tool_call><function=…>` markup into a real call only when it names an offered tool;
|
||
otherwise it hands the markup back as text. The loop read text with no call as the final
|
||
answer, so a 512k-window build that called `web_search` stopped its turn after 3 minutes and
|
||
failed its check. Text recovery that only knows the formats seen so far ends a turn silently on
|
||
the next one: read a model's own call format, and when it names a tool it doesn't have, tell it
|
||
so and name the ones it has, rather than ending.
|
||
|
||
**A view that asked to be scrolled on every frame.** The command palette called `scroll_to_me` on
|
||
the selected row each frame, so the wheel's scroll was pulled back to it at once and the list
|
||
kept jumping to the top (#584). Scroll something into view on the frame it changed (a key, a new
|
||
query), never as a standing request.
|
||
|
||
**A test wrote into the real config folder.** A helper that keeps one id per installation created
|
||
opencode-session under settings::config_dir() when the first request was made, so running the
|
||
tests left a file on the developer's machine and two runs could collide. Anything we persist goes
|
||
through the hub's stores (AiHub::with_stores, Session::isolated), which tests point at a temp
|
||
folder; production code never writes under the real config folder by hand.
|
||
|
||
**A per-probe walk repeated expensive work on the same bytes.** CRC cpio verification was bounded
|
||
inside one archive, but a firmware scan could rediscover overlapping, trailerless archives and
|
||
sum the same payload repeatedly. Count checksum byte visits across the whole scan, poll its
|
||
cancellation callback inside long sums, and distinguish skipped verification from a mismatch.
|
||
Test overlapping candidates with correct checksums and a complete nested archive so a structural
|
||
trailer cannot turn unchecked data into a high-confidence result.
|
||
|
||
**A line cap that cut a JSON-RPC answer in pieces.** MCP over stdio answers one message per line,
|
||
and a server with many richly described tools answers `tools/list` as several megabytes on that
|
||
one line. The process reader capped a partial line at 1 MiB and delivered it anyway, so the answer
|
||
arrived as fragments that never parsed and the plugin failed as a `tools/list` timeout — while the
|
||
same server answered a direct probe in under a second. A cap protects memory, but a JSON-RPC line
|
||
must arrive whole to parse at all: keep the cap at the size the MCP client already allows, and test
|
||
with a multi-megabyte line, not a short one.
|
||
|
||
**A capture budget checked before deduplication changed the replay.** Redaction expanded a repeated
|
||
body, so its second copy was rejected before it could be discarded as a duplicate. Apply aggregate
|
||
budgets only to distinct bodies actually retained. When a budget removes a body, preserve unavailable
|
||
cached 304 responses and deduplicate identical omitted samples.
|
||
|
||
**Two worktrees shared build artifacts during conflict checks.** Concurrent Cargo checks of
|
||
different branches reused the same package output path and one saw the other branch's engine.
|
||
Give every worktree its own `CARGO_TARGET_DIR`, even when one agent manages both. Cleaning
|
||
the shared engine cache then removed 11.6 GiB of compiled artifacts: avoid shared-cache
|
||
cleanup as a recovery shortcut; use an isolated target directory for further verification.
|
||
|
||
**A partial archive listing counted as a complete install.** A GNU-name limit correctly stopped
|
||
an archive parser, but the installer wrote earlier listed files and returned success. Preserve the
|
||
partial inventory for inspection, expose the stop as typed data, and make extraction check it before
|
||
writing any member. Missing end markers need their own compatibility control; `complete: false`
|
||
alone does not identify a refused metadata record.
|
||
|
||
**A roadmap read from the checked-out branch, not from `main`.** `docs/ROADMAP.md` was read while
|
||
an old feature branch was checked out, so a dozen pending lines that `main` had already ticked
|
||
looked open: the SSH builder's environment, the 304 stand-in, prompt pictures, pins' size, the
|
||
OpenCode sweep. Time went into checking each against the code, and a branch was even started for
|
||
work `main` already had. Read the roadmap (and any file you plan to change) from `origin/main`,
|
||
with `git show origin/main:docs/ROADMAP.md` or after checking out the base branch; a pending line
|
||
can also carry "— in PR #245", so check that PR's state, since a closed PR leaves the work open.
|
||
|
||
**Safe recovery paths erased distinct source identities.** Source-map names such as `src//main.js`
|
||
and `src/main.js` became the same safe path, so exclusive creation kept the first file and dropped
|
||
the second. A file can also occupy a later source's parent directory, or arrive after its own
|
||
child. Preserve ordinary output paths and put collisions in an independent input/source-index
|
||
directory; keep complete source identities in the manifest. Check both prefix orders and
|
||
same-named input scripts through the real recovery tool, and retain the writer's symlink and
|
||
exclusive-creation checks after selecting a name.
|
||
|
||
**A settings smoke test clicked before scrolling settled.** An asynchronous refusal added a row,
|
||
then a fixed number of frames left the next action outside the settled layout. Wait for the
|
||
background operation to finish and run immediate repaints after scrolling before clicking;
|
||
otherwise a missed click looks like a backend mutation that never completed.
|