Files
craft-ai-integrations/docs/STATUS.md
Zen Creative Director 6507c6f42c feat(ai-engine-rten): BEN2 + ViTMatte for Remove background (#167)
Adds a built-in rten runner for image.matte: BEN2 finds the subject, and the small and base ViTMatte models matte its edge in tiles. The three published ONNX files are offered in Set up AI at pinned revisions with their sizes and hashes. The recipe sits after InSPyReNet, so the default Remove background offer does not change.
2026-11-11 13:30:50 +00:00

15 KiB
Raw Permalink Blame History

Status: what runs, what is tested, what is open

Three levels, kept apart on purpose. A saved key, a static catalogue, a passing scripted test and a live job prove different things.

Built and tested against scripted servers only

No real hosted service, download host or repository was contacted by these tests.

Area Crate Notes
OpenAI-compatible text and images craft-ai-openai Streaming, tools, structured output; image generate, edit and inpaint. A streamed reply is bounded three ways: per line (1 MiB), total bytes kept (the same as a whole reply, 128 MiB), and events held for one poll (65,536; a stream that passes it without a poll fails the job and the connection is dropped).
FLUX image and video, FLUX.1 Fill craft-ai-bfl FLUX.1 Fill [pro] (flux-pro-1.0-fill) serves image.inpaint for Generative fill: source, mask (white or transparent = fill; sent to BFL as a binary grayscale PNG, white = fill) and prompt, with BFL's documented options range-checked. The result is downloaded inline as soon as it is ready (BFL's links last ten minutes), only from an https *.bfl.ai address and without the key. Every BFL job (FLUX 3 too) is polled at the polling_url BFL returns when it is https on bfl.ai or a subdomain (the key goes nowhere else, no redirects); otherwise at {base}/get_result. A task still "not found" (status or 404) after 10 polls in a row and 60 seconds (BflConfig::not_found_grace) fails the job. BFL has no cancel route, so cancelling an unfinished job is Unsupported. FLUX.1 Expand is not mapped (it takes per-side pixel counts, which image.outpaint does not carry).
Gemini image fill with an AI Studio key craft-ai-gemini The Gemini API (generativelanguage.googleapis.com/v1beta, Interactions endpoint), not Vertex. Serves image.inpaint for Generative fill with Nano Banana 2.1 (default), Pro, 2 and 2 Lite. Gemini takes no mask: the mask is sent as a second picture (opaque, white = fill) with an instruction explaining it, so this is prompt-guided editing and the model may change pixels outside the mask (the app keeps only the masked part). An empty prompt asks for the area to be filled from its surroundings. Output is JPEG at the nearest supported aspect ratio, 1K/2K/4K by the picture's size or image_size. A text-only answer fails the job with the model's words. The key test lists models (GET /models, free). Each job runs on its own thread; cancelling stops waiting but a sent request may still be billed.
Seedance video through BytePlus ModelArk craft-ai-seedance The Volcengine China address is refused.
Gemini images on Vertex express mode craft-ai-vertex An express-mode key is not a full-project Vertex credential and has no free key probe.
ArtCraft Omni API key craft-ai-artcraft Prompt-only image/video and Nano Banana 1/2/Pro mask-guided image editing. Scripted checks cover private uploads, selected model, consent, inline results and refused delivery responses. See Omni Generative Fill for the limited live check and quality gaps. Account or subscription access has no adapter: no official programmatic route is documented. An API key is not evidence that a subscription covers the requests.
Hosted media copies craft-ai-media https only (loopback only for a connection marked local), every redirect re-checked, private addresses refused, size bounded, file type taken from the bytes. A failed copy keeps the original link with a notice. A busy host is retried with a bounded, doubling pause. Provider API calls follow the same rule (craft_ai_core::Backoff): a rate-limited send is tried again, a dropped send never is (the job may already be running, so a second send could charge twice), and a poll that fails for a reason that can pass does not end Job::wait. A cut download resumes on the next try when the host named the file's version (a strong ETag or Last-Modified, sent back as If-Range), and starts over otherwise. A link's expiry is read from S3, Cloud Storage, CloudFront and Azure signatures: a link clearly past it is refused before any request, and a failed copy keeps the link with PayloadHandle::expires_at set and a notice saying when it stops working. Not built: expiry for providers that state a lifetime without signing it into the link, and an input-side upload cache.
Generic ComfyUI workflows craft-ai-comfyui Maps a prompt field; image.inpaint needs mapped LoadImage and LoadImageMask fields. A video.generate workflow may map one LoadImage field as its start image, which every request must then carry. A workflow is offered as a model only after /object_info confirms its nodes and checkpoint names; submission repeats the check. Output stays inline (32 MiB) unless the app gives a directory (1 GiB, local file: handle, app owns cleanup). Targeted cancel is off by default because an older server may treat it as a global interrupt. Imported image.segment workflows map a description and source to a checked, source-sized grayscale mask plus the shared result. Empty masks mean no match; source uploads are remembered for selection jobs. This is not calibrated absence detection: the example Florence graph can find a box for an absent subject. Source and mask images may be inline or a local file: handle. No general parameter mapping yet.
Built-in engines craft-ai-engine, craft-ai-engine-candle, craft-ai-engine-rten, craft-ai-engine-burn Contract, registry and provider are tested with a fake runner and the shared conformance suite. SAM 3 on Candle (point and text prompts, processor, or Metal on macOS) ran on a real photograph and passes the same suite on Meta's checkpoint; the live tests skip without CRAFT_SAM3_WEIGHTS. It reads the Hugging Face checkpoint layout with tokenizer files, not the ComfyUI file. Pictures go in as PNG or JPEG (a JPEG's EXIF turn is applied). No Nvidia (CUDA) build, and the launcher does not download the checkpoint yet. SAM 2.1 on rten (processor only) reads the publisher's two .onnx files, pinned by size and SHA-256, with click and box prompts; a description is refused in plain words. Pictures are bounded by side and pixel count. With the real files in release on one Windows machine: a mask at the picture's size, and a hover through the ordinary job path with a median of about 35 ms after a first look of about 0.7 s (one picture, not a benchmark). SAM 2.1 is the first recipe for Select subject, so the plain built-in offer is SAM 2.1 on rten. No macOS or Linux run, and no app has run it through the shared binding yet. SAM 2.1 on Burn (craft-ai-engine-burn, cargo feature burn, optional, after rten in the recipe) reads weights that craft-ai-burn-setup prepares once on the computer; on one Windows machine with a graphics card its mask matched rten's (best-mask IoU 1.0, one click and one box, one picture, not a benchmark). Given the downloaded encoder.onnx, the runner prepares the saved copy itself on first load (in burn-prepared/ beside the model folder; no progress and no Cancel during that load). No app has run it through the shared binding yet, and there has been no live run on macOS or Linux. InSPyReNet on Burn (inspyrenet, same crate and feature) answers image.matte from the picture alone with a grayscale PNG at the picture's size, at any picture size. Its hand-written network reads the published .safetensors checkpoint (checked tensor for tensor against the author's ckpt_base.pth); on one Windows machine every stage matched transparent-background's PyTorch code (worst relative difference 3.2e-5 on the processor, 2.4e-4 on the card, no pixel changing side), it passes the shared conformance suite, and on a real PNG its mask was byte-identical to the same port inside PhotoCraft. Time per picture was not measured cleanly (the card was shared during the runs). remove-background lists it as its first recipe, built in only (LocalRecipe::comfyui false): Set up AI offers the Burn route and its pinned file, never a ComfyUI model, and scripted tests cover the offer, the plan, the registration and the advice, and an ignored test ran the real file from that route through AiClient for two app ids; the file has not been downloaded through Set up AI, and no real app has mounted it. See docs/BURN-ENGINE.md. BEN2 + ViTMatte on rten (ben2-vitmatte, processor only) is the second built-in recipe for remove-background: BEN2 finds the subject on the whole picture and again in tiles along its contour, then the small and base ViTMatte both matte every edge pixel from a trimap and their alphas are averaged. It takes no parameters. It reads the three publishers' .onnx files as they are, pinned by size and SHA-256, and answers with a grayscale PNG at the picture's size. On one macOS (Apple Silicon) machine: the runner passes the shared conformance suite with the real files; rten's outputs matched ONNX Runtime 1.19 on the same inputs (mean difference 5e-5 for BEN2, 1e-5 and 7e-5 for the two ViTMatte models; worst pixels 0.003, 0.06 and 0.13); a 1600 × 1200 disc took 67 s in release. No larger picture has been timed on rten. No Windows or Linux run, the files have not been downloaded through Set up AI, and no app has run it through the shared binding yet.
First-run setup flow craft-ai-client, craft-ai-egui, craft-ai-features One start button on SetupSession (checks complete files on disk, otherwise downloads; blocker says why it does nothing), a licence tick only where a file's licence needs one, FolderDesk::ensure_default for the suggested folder, ToolDesk::adopt for checked files only, a try-it sample that clicks for built-in selection models and fails a selection that chose nothing, status and picker that agree when a model's file has gone, and engine advice that does not call a measured-quick built-in engine (rten, one machine) slow. Scripted tests and a fake provider only: no real download, no model run through it, no app has mounted it, no macOS or Linux run.
Shared egui widgets craft-ai-egui Source and tests migrated to egui 0.37 with the reviewed storytold egui family at 6d518fc799681571963842f95b55a6b68431b15a and winit at a517b3846ee8457d84ee4975110bfda6797c44dc. The current headless candidate passed 126 tests with 0 failures and 4 ignored doctests under managed job egui037-ai-final-ci-20261025-1331, with formatting that left the worktree unchanged and strict all-target clippy for craft-ai-egui and craft-ai-engine on Linux. Five deterministic kittest wgpu/snapshot goldens are committed and passed in that run. The historical 108-test checkpoint at egui037-ai-regression-mutation-20261024-2205 also recorded a pure-background RGB(27,27,27) to RGB(30,30,30) change failing the expected snapshot with 75,486 changed pixels, and the two compiled connection-panel lifecycle mutants caught before the source and golden hashes were restored. A locked consumer-smoke test build and its one draw test passed with one craft-ai-core identity and the pinned egui family; that rendererless root does not use the winit patch. No app imports the widgets. Native X11/Wayland/Windows behavior, IME/clipboard/focus, physical GPU qualification and performance measurements remain open. Full remote workspace/platform CI also remains open.
Declarative adapter craft-ai-declarative Request/response and queue flows from a data file; prompt-only inputs. No launcher screen to add a spec yet.
Custom connections, model sources, node packs craft-ai-core, craft-ai-comfy-env See docs/custom-connections-and-sources.md. Own-host tokens only, no key sent across a redirect, pickle formats need consent, unverified checksums are flagged.
Model folders craft-ai-core, craft-ai-comfy-env, craft-ai-client A person's own folders of model files, kept as a map (which sub-folders hold which kind) in the shared store, never moved, renamed or deleted. Scanned read only with limits (depth, files, entries, header size), one default for downloads, folders linked from ComfyUI are read only, and the map can be written as a ComfyUI extra-paths file. Not wired yet: downloads and the installed-files scan still use ComfyUI's own folders, and no launcher screen is merged.
Consent, history, shared store, key handling craft-ai-core, craft-ai-native, craft-ai-client Consent is scoped to app, connection, destination and capability, per kind of data, and may expire. craft-ai-egui offers the original prompt and a timed prompt through consent_prompt_with_terms; no app has mounted it yet.

Run for real

Only the local SAM 3 clients (Sam3Point, Sam3Session): point, box, text, candidate masks, grounding instances and prepared hover sessions, through a headless ComfyUI on one Windows machine. The live-test evidence is not kept in this repository. This does not make the generic ComfyUI image or video workflows live-tested.

Not verified

  • Hosted providers beyond the limited Nano Banana 2 check described in Omni Generative Fill. That check is not output-quality acceptance.
  • TLS against an intercepting proxy or antivirus HTTPS scanner. craft-ai-http trusts the platform's certificate store, so such software can read traffic, API keys included, and there is no switch back to bundled roots.
  • macOS (CI runs the tests on Linux and Windows), and the macOS and Linux keychains. The real keychain test is #[ignore]d; it passed by hand against Windows Credential Manager on 10/11/2026.
  • Model download from Hugging Face and automatic node-pack installation against the real services.
  • The built-in recommended-models catalogue (catalog.json) is empty. Which models to recommend per tool is a curation decision, so "Recommended" never appears until someone makes it.
  • App adoption beyond the limited local PhotoCraft Omni check. Document pixel preservation, saved-result integrity and undo/redo were not verified there. A route-resolution test or a Ready badge is not inference adoption.
  • The keychain is no longer called while the shared file lock is held, but how long a slow real keychain takes is unmeasured.
  • The encrypted-file fallback on a machine with no keychain. standard() now layers the environment, the OS keychain and the encrypted file, which stays locked until a host calls standard_with_passphrase; passphrase_need() says when to ask. Scripted tests stand in for a missing keychain. No host asks for the passphrase yet, and no bare Linux session has run it.

Design limits

  • The runtime is in-process: apps link craft-ai-client and run adapters themselves. A daemon with IPC between apps and the adapters has been suggested by the maintainers and is a separate discussion; no wire format is defined.
  • A registered tool is a declaration, not installed code. No dynamic adapter registry for arbitrary user code, no general tool executor and no automatic model installation.
  • Interactive negative boxes are ignored, grounding exemplar boxes are not strict regions, hover JPEGs are not exact masks, candidate order is not a semantic label, and refinement is not guaranteed to improve quality. Runtime catalogue entries stay executable-only.