Files
craft-ai-integrations/docs/STATUS.md
T
JeremyandFredApps be236d570b feat(ai-client): keep a run record as evidence of a real run (#177)
* feat(ai-client): keep a run record as evidence of a real run

AI-17, first slice. `RunRecord` turns a finished job and its plan into one
JSON line: app, tool, capability, provider, connection, model, destination,
the kinds and sizes sent (as the consent screen shows them), parameter names,
start time and duration, outcome, error kind and wording, notices, usage, and
each result's type, size and SHA-256. It keeps no prompt, no result bytes and
no link, since a hosted link can carry a signed token. `to_json_line` refuses
a line that contains any secret it is given, so a key that slipped into an
error message never reaches the file.

The `live_openai` example takes `--record <file>` and appends the line for
its run. No live run was made: tests use a scripted localhost server and the
mock provider. Deliberate-bug group `rr` (10 cases) catches all 10.

* docs(ai-client): say a timed-out wait leaves no run record

Job::wait returns an error on a timeout, so RunOutcome::Unfinished only
comes from a Finished the caller built from a job that had not ended.
Reword the variant's doc and the test that covers it.

* fix(ai-client): clean run-record messages and catch escaped keys

Error and notice text is cleaned when the record is built: the secrets
given to RunRecord::new, the request's own text inputs and any word that
looks like an address are replaced. The record never holds a key, so
serialising it by any route is safe. to_json_line also refuses a line
holding a secret in its JSON-escaped form, so a key with a quote or a
backslash cannot slip past the check.

* fix(ai-client): remove a key inside the request text with the whole text

scrub replaced secrets before it looked for the request text, so a key
inside the prompt broke the prompt's match and left the words around it in
the record. Secrets and request texts are now all found in the original
message and overlapping matches are removed together. The module doc no
longer claims that serialising a record by any route is safe: only its
messages are cleaned, so records are written with to_json_line.

* fix(ai-client): remove overlapping repeats of the request text

match_indices skips overlapping occurrences of the same text, so request
text such as "private private" inside "private private private" left part
of itself behind. Every occurrence is now found, overlapping ones included,
and the spans merge as before.

* fix(ai-client): leak-check the run record and saved image in live_openai

The final leak check only read folders, so the --record file and the
image written by 'image <out.png>' were never read back. It now reads
single files as well as folders, and both are added to the list.

* fix(ai-client): clean parameter text and escaped forms from run records

Error and notice text is now also cleaned of the request's text
parameter values, nested ones included, and of every key and text in
its JSON-escaped form. live_openai refuses a record path that is also
the image output, and its leak check fails when a written file cannot
be read instead of reporting clean. Group rr gains four cases; 24 of
24 caught.

* fix(ai-client): find links in the original text of a run-record message

Links were removed after keys and request text, so request text inside
a link split the address and kept the rest of it, signed token
included. Link spans are now found in the original text and merged with
the others; a merged stretch is named key, then link, then request
text. Also collapses the nested if Clippy flagged in live_openai.
Group rr: 24 of 24 caught.

* docs(ai-client): keep two recorded live runs as evidence

live_openai gains an omni mode (one nano_banana image from a fixed
prompt through the saved ArtCraft Omni connection) and --connection to
pick one saved connection by id. Omni has no free key test, so its
saved model list is used as it stands.

docs/evidence/live-runs.jsonl keeps two records from 11/11/2026: an
OpenAI text run refused with 429 (no API credit) and an Omni image run
that succeeded with one image/png link. STATUS.md says what each shows
and what it does not.

* fix(ai-client): clean returned text and escaped links from run records

Error and notice text is now also cleaned of what the run returned (each
text result and the whole streamed text), read as [result text removed],
and of links written with JSON-escaped slashes. The live_openai docs no
longer promise a hash for a returned link. Group rr: 27 of 27 caught.

* fix(ai-client): protect the shared store from live example outputs

* fix(ai-client): keep record lines whole and resolve parent folders in live_openai

A record appended to a file that ends without a newline now gets one
first, so two records never share a line; append_line moves to its own
module with tests. The output preflight compares names with each parent
folder resolved, so a linked folder or '..' cannot hide that the record
and the image are one file.

* fix(ai-client): keep only the adapter's own words of a run record's messages

Two ways a record could still hold what the person sent or received:
adapters cut long upstream messages at 300 or 360 characters, so a prompt
echoed back arrived cut part-way and never matched whole; and a failed
Gemini image run put the model's own text reply in its error, which was
never an output and so never removed.

Error and notice text now keeps only the adapter's lead. What follows the
first ": " is where adapters quote the service or model, so it becomes
"[detail removed]", and an error records its length and hash. Before the
cut, any stretch of 16 or more characters taken from a request or returned
text is removed too. The record version goes to 2, since `message` changed
meaning and `detail` is new.

The example's record file is now written with one write per line, so two
runs appending at once cannot split a record.

* fix(ai-client): recheck image aliases after recording runs

* test(ai-client): keep the locked-output check true on every Windows machine

Whether a file held with no sharing still has a readable identity differs
between Windows installs: it does on some, and the CI runners refuse it. The
test asserted one outcome, so it passed locally and failed in CI. It now
checks the invariant that holds either way: an unreadable identity is refused
up front, a readable one fails at the write, and the locked file and the
shared store are never changed.

---------

Co-authored-by: FredApps <FredApps@users.noreply.github.com>
2026-11-13 22:01:40 +01:00

17 KiB
Raw Blame History

Status: what runs, what is tested, what is open

Three levels, kept apart on purpose. A saved key, a static catalogue, a passing scripted test and a live job prove different things.

Built and tested against scripted servers only

No real hosted service, download host or repository was contacted by these tests.

Area Crate Notes
OpenAI-compatible text and images craft-ai-openai Streaming, tools, structured output; image generate, edit and inpaint. A streamed reply is bounded three ways: per line (1 MiB), total bytes kept (the same as a whole reply, 128 MiB), and events held for one poll (65,536; a stream that passes it without a poll fails the job and the connection is dropped).
FLUX image and video, FLUX.1 Fill craft-ai-bfl FLUX.1 Fill [pro] (flux-pro-1.0-fill) serves image.inpaint for Generative fill: source, mask (white or transparent = fill; sent to BFL as a binary grayscale PNG, white = fill) and prompt, with BFL's documented options range-checked. The result is downloaded inline as soon as it is ready (BFL's links last ten minutes), only from an https *.bfl.ai address and without the key. Every BFL job (FLUX 3 too) is polled at the polling_url BFL returns when it is https on bfl.ai or a subdomain (the key goes nowhere else, no redirects); otherwise at {base}/get_result. A task still "not found" (status or 404) after 10 polls in a row and 60 seconds (BflConfig::not_found_grace) fails the job. BFL has no cancel route, so cancelling an unfinished job is Unsupported. FLUX.1 Expand is not mapped (it takes per-side pixel counts, which image.outpaint does not carry).
Gemini image fill with an AI Studio key craft-ai-gemini The Gemini API (generativelanguage.googleapis.com/v1beta, Interactions endpoint), not Vertex. Serves image.inpaint for Generative fill with Nano Banana 2.1 (default), Pro, 2 and 2 Lite. Gemini takes no mask: the mask is sent as a second picture (opaque, white = fill) with an instruction explaining it, so this is prompt-guided editing and the model may change pixels outside the mask (the app keeps only the masked part). An empty prompt asks for the area to be filled from its surroundings. Output is JPEG at the nearest supported aspect ratio, 1K/2K/4K by the picture's size or image_size. A text-only answer fails the job with the model's words. The key test lists models (GET /models, free). Each job runs on its own thread; cancelling stops waiting but a sent request may still be billed.
Seedance video through BytePlus ModelArk craft-ai-seedance The Volcengine China address is refused.
Gemini images on Vertex express mode craft-ai-vertex An express-mode key is not a full-project Vertex credential and has no free key probe.
ArtCraft Omni API key craft-ai-artcraft Prompt-only image/video and Nano Banana 1/2/Pro mask-guided image editing. Scripted checks cover private uploads, selected model, consent, inline results and refused delivery responses. See Omni Generative Fill for the limited live check and quality gaps. Account or subscription access has no adapter: no official programmatic route is documented. An API key is not evidence that a subscription covers the requests.
Hosted media copies craft-ai-media https only (loopback only for a connection marked local), every redirect re-checked, private addresses refused, size bounded, file type taken from the bytes. A failed copy keeps the original link with a notice. A busy host is retried with a bounded, doubling pause. Provider API calls follow the same rule (craft_ai_core::Backoff): a rate-limited send is tried again, a dropped send never is (the job may already be running, so a second send could charge twice), and a poll that fails for a reason that can pass does not end Job::wait. A cut download resumes on the next try when the host named the file's version (a strong ETag or Last-Modified, sent back as If-Range), and starts over otherwise. A link's expiry is read from S3, Cloud Storage, CloudFront and Azure signatures: a link clearly past it is refused before any request, and a failed copy keeps the link with PayloadHandle::expires_at set and a notice saying when it stops working. Not built: expiry for providers that state a lifetime without signing it into the link, and an input-side upload cache.
Generic ComfyUI workflows craft-ai-comfyui Maps a prompt field; image.inpaint needs mapped LoadImage and LoadImageMask fields. A video.generate workflow may map one LoadImage field as its start image, which every request must then carry. A workflow is offered as a model only after /object_info confirms its nodes and checkpoint names; submission repeats the check. Output stays inline (32 MiB) unless the app gives a directory (1 GiB, local file: handle, app owns cleanup). Targeted cancel is off by default because an older server may treat it as a global interrupt. Imported image.segment workflows map a description and source to a checked, source-sized grayscale mask plus the shared result. Empty masks mean no match; source uploads are remembered for selection jobs. This is not calibrated absence detection: the example Florence graph can find a box for an absent subject. Source and mask images may be inline or a local file: handle. No general parameter mapping yet.
Built-in engines craft-ai-engine, craft-ai-engine-candle, craft-ai-engine-rten, craft-ai-engine-burn Contract, registry and provider are tested with a fake runner and the shared conformance suite. SAM 3 on Candle (point and text prompts, processor, or Metal on macOS) ran on a real photograph and passes the same suite on Meta's checkpoint; the live tests skip without CRAFT_SAM3_WEIGHTS. It reads the Hugging Face checkpoint layout with tokenizer files, not the ComfyUI file. Pictures go in as PNG or JPEG (a JPEG's EXIF turn is applied). No Nvidia (CUDA) build, and the launcher does not download the checkpoint yet. SAM 2.1 on rten (processor only) reads the publisher's two .onnx files, pinned by size and SHA-256, with click and box prompts; a description is refused in plain words. Pictures are bounded by side and pixel count. With the real files in release on one Windows machine: a mask at the picture's size, and a hover through the ordinary job path with a median of about 35 ms after a first look of about 0.7 s (one picture, not a benchmark). SAM 2.1 is the first recipe for Select subject, so the plain built-in offer is SAM 2.1 on rten. No macOS or Linux run, and no app has run it through the shared binding yet. SAM 2.1 on Burn (craft-ai-engine-burn, cargo feature burn, optional, after rten in the recipe) reads weights that craft-ai-burn-setup prepares once on the computer; on one Windows machine with a graphics card its mask matched rten's (best-mask IoU 1.0, one click and one box, one picture, not a benchmark). Given the downloaded encoder.onnx, the runner prepares the saved copy itself on first load (in burn-prepared/ beside the model folder; no progress and no Cancel during that load). No app has run it through the shared binding yet, and there has been no live run on macOS or Linux. InSPyReNet on Burn (inspyrenet, same crate and feature) answers image.matte from the picture alone with a grayscale PNG at the picture's size, at any picture size. Its hand-written network reads the published .safetensors checkpoint (checked tensor for tensor against the author's ckpt_base.pth); on one Windows machine every stage matched transparent-background's PyTorch code (worst relative difference 3.2e-5 on the processor, 2.4e-4 on the card, no pixel changing side), it passes the shared conformance suite, and on a real PNG its mask was byte-identical to the same port inside PhotoCraft. Time per picture was not measured cleanly (the card was shared during the runs). remove-background lists it as its first recipe, built in only (LocalRecipe::comfyui false): Set up AI offers the Burn route and its pinned file, never a ComfyUI model, and scripted tests cover the offer, the plan, the registration and the advice, and an ignored test ran the real file from that route through AiClient for two app ids; the file has not been downloaded through Set up AI, and no real app has mounted it. See docs/BURN-ENGINE.md. BEN2 + ViTMatte on rten (ben2-vitmatte, processor only) is the second built-in recipe for remove-background: BEN2 finds the subject on the whole picture and again in tiles along its contour, then the small and base ViTMatte both matte every edge pixel from a trimap and their alphas are averaged. It takes no parameters. It reads the three publishers' .onnx files as they are, pinned by size and SHA-256, and answers with a grayscale PNG at the picture's size. On one macOS (Apple Silicon) machine: the runner passes the shared conformance suite with the real files; rten's outputs matched ONNX Runtime 1.19 on the same inputs (mean difference 5e-5 for BEN2, 1e-5 and 7e-5 for the two ViTMatte models; worst pixels 0.003, 0.06 and 0.13); a 1600 × 1200 disc took 67 s in release. No larger picture has been timed on rten. No Windows or Linux run, the files have not been downloaded through Set up AI, and no app has run it through the shared binding yet.
First-run setup flow craft-ai-client, craft-ai-egui, craft-ai-features One start button on SetupSession (checks complete files on disk, otherwise downloads; blocker says why it does nothing), a licence tick only where a file's licence needs one, FolderDesk::ensure_default for the suggested folder, ToolDesk::adopt for checked files only, a try-it sample that clicks for built-in selection models and fails a selection that chose nothing, status and picker that agree when a model's file has gone, and engine advice that does not call a measured-quick built-in engine (rten, one machine) slow. Scripted tests and a fake provider only: no real download, no model run through it, no app has mounted it, no macOS or Linux run.
Shared egui widgets craft-ai-egui Source and tests migrated to egui 0.37 with the reviewed storytold egui family at 6d518fc799681571963842f95b55a6b68431b15a and winit at a517b3846ee8457d84ee4975110bfda6797c44dc. The current headless candidate passed 126 tests with 0 failures and 4 ignored doctests under managed job egui037-ai-final-ci-20261025-1331, with formatting that left the worktree unchanged and strict all-target clippy for craft-ai-egui and craft-ai-engine on Linux. Five deterministic kittest wgpu/snapshot goldens are committed and passed in that run. The historical 108-test checkpoint at egui037-ai-regression-mutation-20261024-2205 also recorded a pure-background RGB(27,27,27) to RGB(30,30,30) change failing the expected snapshot with 75,486 changed pixels, and the two compiled connection-panel lifecycle mutants caught before the source and golden hashes were restored. A locked consumer-smoke test build and its one draw test passed with one craft-ai-core identity and the pinned egui family; that rendererless root does not use the winit patch. No app imports the widgets. Native X11/Wayland/Windows behavior, IME/clipboard/focus, physical GPU qualification and performance measurements remain open. Full remote workspace/platform CI also remains open.
Declarative adapter craft-ai-declarative Request/response and queue flows from a data file; prompt-only inputs. No launcher screen to add a spec yet.
Custom connections, model sources, node packs craft-ai-core, craft-ai-comfy-env See docs/custom-connections-and-sources.md. Own-host tokens only, no key sent across a redirect, pickle formats need consent, unverified checksums are flagged.
Model folders craft-ai-core, craft-ai-comfy-env, craft-ai-client A person's own folders of model files, kept as a map (which sub-folders hold which kind) in the shared store, never moved, renamed or deleted. Scanned read only with limits (depth, files, entries, header size), one default for downloads, folders linked from ComfyUI are read only, and the map can be written as a ComfyUI extra-paths file. Not wired yet: downloads and the installed-files scan still use ComfyUI's own folders, and no launcher screen is merged.
Consent, history, shared store, key handling craft-ai-core, craft-ai-native, craft-ai-client Consent is scoped to app, connection, destination and capability, per kind of data, and may expire. craft-ai-egui offers the original prompt and a timed prompt through consent_prompt_with_terms; no app has mounted it yet.
Agents switch and app list craft-ai-agents Unit tests: the list written, read back, replaced and removed; an app allowed only while the list names it; broken lists count as off. Used by the gateway; the launcher and PhotoCraft adopt it in storytold/craft-launcher#1432 and storytold/photocraft#13396.
MCP gateway to the Craft apps craft-ai-mcp Scripted tests drive in-process fake app servers (handshake, paging, timeouts, restarts, pings, bad arguments). Run once on Windows against a photocraft-cli debug build from PhotoCraft's main (2026-11-12): search, describe and doc_new in the open window through mcp --live, and a window screenshot back. No other app run through it; no real MCP client (Claude Desktop, Cursor) started it.

Run for real

Only the local SAM 3 clients (Sam3Point, Sam3Session): point, box, text, candidate masks, grounding instances and prepared hover sessions, through a headless ComfyUI on one Windows machine. The live-test evidence is not kept in this repository. This does not make the generic ComfyUI image or video workflows live-tested.

A real run can now leave evidence: craft_ai_client::RunRecord turns a finished job into one JSON line (where it went, the kinds and sizes sent, the outcome, and each result's type, size and SHA-256, with no prompt, result bytes or link; error and notice text is cleaned of the key, the request text, text parameter values and links, also in JSON-escaped form), and refuses a line that still contains the key. The live_openai example writes one with --record <file>, for OpenAI or, with omni, ArtCraft Omni.

Two recorded live runs through AiClient (plan, consent, start, wait) are kept in evidence/live-runs.jsonl, both on 11 November 2026:

  • OpenAI gpt-4o-mini text: the service refused it with 429 (no API credit on the account). The record shows the failure as quota, with the billing link in the message removed.
  • ArtCraft Omni nano_banana image from a prompt: succeeded in 6.8 s and returned one image/png link. The record keeps only that type; Omni stated no size, and the image itself was not downloaded, so no hash of its bytes exists. The image's content was not checked.

In both runs the person's real store was left unchanged and no file written held the key.

Not verified

  • Hosted providers beyond the limited Nano Banana 2 check described in Omni Generative Fill. That check is not output-quality acceptance.
  • TLS against an intercepting proxy or antivirus HTTPS scanner. craft-ai-http trusts the platform's certificate store, so such software can read traffic, API keys included, and there is no switch back to bundled roots.
  • macOS (CI runs the tests on Linux and Windows), and the macOS and Linux keychains. The real keychain test is #[ignore]d; it passed by hand against Windows Credential Manager on 10/11/2026.
  • Model download from Hugging Face and automatic node-pack installation against the real services.
  • The built-in recommended-models catalogue (catalog.json) is empty. Which models to recommend per tool is a curation decision, so "Recommended" never appears until someone makes it.
  • App adoption beyond the limited local PhotoCraft Omni check. Document pixel preservation, saved-result integrity and undo/redo were not verified there. A route-resolution test or a Ready badge is not inference adoption.
  • The keychain is no longer called while the shared file lock is held, but how long a slow real keychain takes is unmeasured.
  • The encrypted-file fallback on a machine with no keychain. standard() now layers the environment, the OS keychain and the encrypted file, which stays locked until a host calls standard_with_passphrase; passphrase_need() says when to ask. Scripted tests stand in for a missing keychain. No host asks for the passphrase yet, and no bare Linux session has run it.

Design limits

  • The runtime is in-process: apps link craft-ai-client and run adapters themselves. A daemon with IPC between apps and the adapters has been suggested by the maintainers and is a separate discussion; no wire format is defined.
  • A registered tool is a declaration, not installed code. No dynamic adapter registry for arbitrary user code, no general tool executor and no automatic model installation.
  • Interactive negative boxes are ignored, grounding exemplar boxes are not strict regions, hover JPEGs are not exact masks, candidate order is not a semantic label, and refinement is not guaranteed to improve quality. Runtime catalogue entries stay executable-only.