Files
craft-ai-integrations/docs/evidence
JeremyandFredApps be236d570b feat(ai-client): keep a run record as evidence of a real run (#177)
* feat(ai-client): keep a run record as evidence of a real run

AI-17, first slice. `RunRecord` turns a finished job and its plan into one
JSON line: app, tool, capability, provider, connection, model, destination,
the kinds and sizes sent (as the consent screen shows them), parameter names,
start time and duration, outcome, error kind and wording, notices, usage, and
each result's type, size and SHA-256. It keeps no prompt, no result bytes and
no link, since a hosted link can carry a signed token. `to_json_line` refuses
a line that contains any secret it is given, so a key that slipped into an
error message never reaches the file.

The `live_openai` example takes `--record <file>` and appends the line for
its run. No live run was made: tests use a scripted localhost server and the
mock provider. Deliberate-bug group `rr` (10 cases) catches all 10.

* docs(ai-client): say a timed-out wait leaves no run record

Job::wait returns an error on a timeout, so RunOutcome::Unfinished only
comes from a Finished the caller built from a job that had not ended.
Reword the variant's doc and the test that covers it.

* fix(ai-client): clean run-record messages and catch escaped keys

Error and notice text is cleaned when the record is built: the secrets
given to RunRecord::new, the request's own text inputs and any word that
looks like an address are replaced. The record never holds a key, so
serialising it by any route is safe. to_json_line also refuses a line
holding a secret in its JSON-escaped form, so a key with a quote or a
backslash cannot slip past the check.

* fix(ai-client): remove a key inside the request text with the whole text

scrub replaced secrets before it looked for the request text, so a key
inside the prompt broke the prompt's match and left the words around it in
the record. Secrets and request texts are now all found in the original
message and overlapping matches are removed together. The module doc no
longer claims that serialising a record by any route is safe: only its
messages are cleaned, so records are written with to_json_line.

* fix(ai-client): remove overlapping repeats of the request text

match_indices skips overlapping occurrences of the same text, so request
text such as "private private" inside "private private private" left part
of itself behind. Every occurrence is now found, overlapping ones included,
and the spans merge as before.

* fix(ai-client): leak-check the run record and saved image in live_openai

The final leak check only read folders, so the --record file and the
image written by 'image <out.png>' were never read back. It now reads
single files as well as folders, and both are added to the list.

* fix(ai-client): clean parameter text and escaped forms from run records

Error and notice text is now also cleaned of the request's text
parameter values, nested ones included, and of every key and text in
its JSON-escaped form. live_openai refuses a record path that is also
the image output, and its leak check fails when a written file cannot
be read instead of reporting clean. Group rr gains four cases; 24 of
24 caught.

* fix(ai-client): find links in the original text of a run-record message

Links were removed after keys and request text, so request text inside
a link split the address and kept the rest of it, signed token
included. Link spans are now found in the original text and merged with
the others; a merged stretch is named key, then link, then request
text. Also collapses the nested if Clippy flagged in live_openai.
Group rr: 24 of 24 caught.

* docs(ai-client): keep two recorded live runs as evidence

live_openai gains an omni mode (one nano_banana image from a fixed
prompt through the saved ArtCraft Omni connection) and --connection to
pick one saved connection by id. Omni has no free key test, so its
saved model list is used as it stands.

docs/evidence/live-runs.jsonl keeps two records from 11/11/2026: an
OpenAI text run refused with 429 (no API credit) and an Omni image run
that succeeded with one image/png link. STATUS.md says what each shows
and what it does not.

* fix(ai-client): clean returned text and escaped links from run records

Error and notice text is now also cleaned of what the run returned (each
text result and the whole streamed text), read as [result text removed],
and of links written with JSON-escaped slashes. The live_openai docs no
longer promise a hash for a returned link. Group rr: 27 of 27 caught.

* fix(ai-client): protect the shared store from live example outputs

* fix(ai-client): keep record lines whole and resolve parent folders in live_openai

A record appended to a file that ends without a newline now gets one
first, so two records never share a line; append_line moves to its own
module with tests. The output preflight compares names with each parent
folder resolved, so a linked folder or '..' cannot hide that the record
and the image are one file.

* fix(ai-client): keep only the adapter's own words of a run record's messages

Two ways a record could still hold what the person sent or received:
adapters cut long upstream messages at 300 or 360 characters, so a prompt
echoed back arrived cut part-way and never matched whole; and a failed
Gemini image run put the model's own text reply in its error, which was
never an output and so never removed.

Error and notice text now keeps only the adapter's lead. What follows the
first ": " is where adapters quote the service or model, so it becomes
"[detail removed]", and an error records its length and hash. Before the
cut, any stretch of 16 or more characters taken from a request or returned
text is removed too. The record version goes to 2, since `message` changed
meaning and `detail` is new.

The example's record file is now written with one write per line, so two
runs appending at once cannot split a record.

* fix(ai-client): recheck image aliases after recording runs

* test(ai-client): keep the locked-output check true on every Windows machine

Whether a file held with no sharing still has a readable identity differs
between Windows installs: it does on some, and the CI runners refuse it. The
test asserted one outcome, so it passed locally and failed in CI. It now
checks the invariant that holds either way: an unreadable identity is refused
up front, a readable one fails at the write, and the locked file and the
shared store are never changed.

---------

Co-authored-by: FredApps <FredApps@users.noreply.github.com>
2026-11-13 22:01:40 +01:00
..