Files
craft-ai-integrations/tools/mutation/mutation_engine_memory.py
T
Jeremy fdf02b3b47 feat(ai-engine, ai-client): show, unload and limit models kept in memory (#179)
* feat(ai-engine): show, unload and limit models kept in memory

The built-in engine kept every model it loaded until the app closed, with no way to see or free
that memory. `EngineProvider` now:

- lists loaded models with roughly what each holds (`loaded_models`); a runner reports its own
  figure through `LoadedModel::memory_bytes`, else the weights file size stands in, marked as
  an estimate, and the figure is read again after each run;
- unloads one model or all (`unload`); a model in use finishes its job and is let go of as it
  leaves the slot, so nothing is interrupted and no request is lost;
- unloads models idle for a given time (`unload_idle`), a call the app makes, with no timer;
- with `with_free_memory`, refuses in plain words a load that will not fit, before anything
  loads (`ModelRunner::memory_needed`, default the file size).

Deliberate-bug group `mm`: 13 cases, all caught.

Part of #178. Reading free memory per platform and the AiClient wiring follow separately.

* fix(ai-engine): close the unload hand-off window and guard the free-memory reader

- A job now lets go of the model's slot while it still holds the residency lock, so an unload
  can no longer see a job that has left but still holds the slot and defer the unload to the
  next job.
- A free-memory reader that panics fails that job with an internal error instead of unwinding
  through the worker, which left the job unfinished.
- The fake runner can be held inside its run, so tests that look at a running job no longer
  depend on timing.

Deliberate-bug group `mm`: 14 of 14 caught.

* feat(ai-client): list and free models the built-in engines keep in memory

Apps reach the built-in engines through AiClient, which had no way to see or free the models
they keep loaded. The factory now keeps each engine's provider cache by engine name, and the
client offers:

- `loaded_models()`: each loaded model with its engine, connection and rough size;
- `unload_model(connection, model)`, `unload_all_models()` and `unload_idle_models(time)`;
- `with_free_memory(read)`, which reaches every engine provider at load time, including
  providers built before the reader was set, and survives `with_factory` in either order.

Registering or unregistering an engine name drops its cache, so a replaced engine's models are
no longer listed. Deliberate-bug group `cm`: 6 of 6 caught.

Part of #178. Builds on #179.

* fix(ai-client): keep a replaced engine provider listed and freeable

When a connection changed while a job still held its old provider, the cache replaced it and
the old model dropped out of `loaded_models()` and could not be freed. A replaced provider is
now asked to unload at once (its running job finishes first) and stays listed until its models
are gone.

The cache stays shared by clones of one factory, so clients built from it share one loaded copy
instead of loading the weights twice. The doc comment on `loaded_models` now says so, and a test
pins it. Deliberate-bug group `cm`: 8 of 8 caught.

* fix(ai-engine, ai-client): a replaced provider keeps nothing its queued jobs load

A job queued on a provider before its connection changed could load the model after the
client replaced that provider, leaving a model nobody could list or free. `EngineProvider::retire`
now makes every job let go of its model as it finishes, and the client keeps a retired provider
listed while it is busy or still holds a model.

The mid-job test now uses different file sizes for the old and new model, so it tells them
apart. Deliberate-bug groups: `mm` 15 of 15, `cm` 8 of 8.

* fix(ai-client): keep a replaced provider listed throughout the swap

The replaced provider now moves to the retiring list before the cache lock is released, so a
listing taken during the swap never misses it. A new test changes the connection while the old
job is still loading and a second job waits behind it: both old jobs finish and let go of what
they loaded, and only the new model is left to free. The fake runner can now pause a load.

Deliberate-bug group `cm`: 9 of 9 caught.

* fix(ai-client): keep a retired provider listed while anything still holds it

A caller that got a provider just before its connection changed could submit through it after
a listing had already dropped the retired provider, so the model that job loaded was out of
sight while it ran. A retired provider is now kept while any other reference to it exists, as
well as while it is busy or holds a model. A test holds the old provider across the swap, lists,
then submits through it and checks the model is listed and freeable.

Deliberate-bug group `cm`: 10 of 10 caught.

* test(ai-client): drop an unused import from the loaded-models tests

* fix(ai-engine, ai-client): set memory aside for loads under way

Two loads judged at once both counted the same free memory. Each admitted
load now sets its need aside until its load ends, and every built-in engine
of a client shares one count. A replaced provider is retired after the
cache lock is released, and a provider replaced during a listing is listed
once.
2026-11-14 16:05:37 +00:00

82 lines
6.5 KiB
Python

"""Deliberate-bug cases for group `mm`: models kept in memory by the built-in engine provider
(`ai-engine/src/memory.rs`, `provider.rs`). An unload never interrupts a run and is never lost,
an idle sweep leaves busy and recent models alone, one model's unload leaves the others, the size
shown follows what the model reports, and a load that will not fit is refused before anything
loads.
Loaded by mutation_check.py. Each case breaks one rule and names the test that should notice.
Format: (name, file, old, new, package, test target, test filter).
"""
M = 'crates/ai-engine/src/memory.rs'
P = 'crates/ai-engine/src/provider.rs'
E = 'craft-ai-engine'
BUSY = (
'Err(TryLockError::WouldBlock) => {\n'
' if idle.is_some() {\n'
' return Outcome::Untouched;\n'
' }\n'
' state.unload_requested = true;\n'
' return Outcome::AfterRun;\n'
' }'
)
IDLE_SKIP = (
' if idle.is_some() {\n'
' return Outcome::Untouched;\n'
' }\n'
)
AT_LOAD = ' Some(bytes) => self.residency.loaded(bytes, false),\n'
AFTER_RUN = (
' if let Some(bytes) = reported(model.as_ref()) {\n'
' self.residency.loaded(bytes, false);\n'
' }\n'
)
CHECK = ' let admitted = guarded(|| admission.admit(&self.model.label, needed, free))?;\n'
LOAD_START = ' if slot.is_none() {\n let needed'
MM = [
('unload-request-ignored-on-leave', M, 'let taken = if discard || state.unload_requested {', 'let taken = if discard {', E, 'memory', 'in_use_is_unloaded_when_its_job_finishes'),
('busy-slot-taken-as-free', M, BUSY, 'Err(TryLockError::WouldBlock) => return Outcome::Untouched,', E, 'lib', 'unload_during_a_run_waits'),
('request-lost-before-enter', M, BUSY, BUSY.replace(' state.unload_requested = true;\n', ''), E, 'lib', 'between_taking_the_slot_and_entering'),
('idle-sweep-takes-recent-models', M, 'if state.held.is_none() || !rested {', 'if state.held.is_none() {', E, 'memory', 'idle_unload_frees_only_models_rested'),
('idle-sweep-marks-busy-models', M, IDLE_SKIP, '', E, 'memory', 'idle_unload_leaves_a_model_in_use_alone'),
('unload-leaves-model-listed', M, ' state.held = None;\n state.unload_requested = false;\n held.take()', ' state.unload_requested = false;\n held.take()', E, 'memory', 'unloading_frees_the_model'),
('unload-one-takes-all', P, 'model.is_none_or(|m| &e.model.id == m)', 'true', E, 'memory', 'unloading_one_model_leaves_the_others'),
('memory-check-skipped', P, CHECK, ' let admitted = ();\n', E, 'memory', 'will_not_fit_is_refused'),
('reader-panic-not-guarded', P, CHECK, ' let admitted = admission.admit(&self.model.label, needed, free)?;\n', E, 'memory', 'reader_that_panics'),
('memory-check-on-every-run', P, LOAD_START, ' drop(admission.admit(&self.model.label, Some(1), free)?);\n' + LOAD_START, E, 'memory', 'runs_again_without_a_memory_check'),
('admitted-load-sets-nothing-aside', M, ' *set_aside = set_aside.saturating_add(bytes);\n', '', E, 'memory', 'another_load_has_been_promised'),
('set-aside-ignored-when-judging', M, '.map(|bytes| bytes.saturating_sub(*set_aside));', ';', E, 'memory', 'another_load_has_been_promised'),
('share-never-given-back', M, ' *set_aside = set_aside.saturating_sub(self.bytes);\n', '', E, 'memory', 'another_load_has_been_promised'),
('admission-not-shared', P, ' self.admission = admission;\n', '', E, 'memory', 'sharing_an_admission'),
('fit-compared-the-wrong-way', M, 'if needed <= free {', 'if needed >= free {', E, 'lib', 'fits_or_cannot_be_judged'),
('reported-size-ignored-at-load', P, AT_LOAD, ' Some(_) => self.residency.loaded(needed.unwrap_or_default(), true),\n', E, 'memory', 'reports_its_memory'),
('size-not-read-again-after-a-run', P, AFTER_RUN, '', E, 'memory', 'grows_between_runs'),
('retired-provider-keeps-models', P, ' || retired.load(Ordering::Acquire);', ' || false;', E, 'memory', 'retired_provider_keeps_nothing'),
('bad-model-kept-after-a-crash', P, 'let discard = matches!(&result, Err(e) if e.kind == ErrorKind::Internal)\n', 'let discard = false\n', E, 'provider', 'panicking_runner_fails_the_job_and_is_loaded_afresh'),
]
# Group `cm`: the same, through the app's client (`ai-client/src/engines.rs`, `factory.rs`,
# `client.rs`). Unloading one model needs its connection, a replaced engine's models are not
# listed, and a free-memory reader applies whichever order the client and factory are built in.
CE = 'crates/ai-client/src/engines.rs'
CF = 'crates/ai-client/src/factory.rs'
CC = 'crates/ai-client/src/client.rs'
C = 'craft-ai-client'
T = 'loaded_models'
CM = [
('unload-one-ignores-connection', CE, '(Some((connection, model)), None) if connection.as_str() == id => {', '(Some((_, model)), None) => {', C, T, 'needs_its_connection'),
('label-not-listed', CE, 'connection_label: label.clone(),', 'connection_label: String::new(),', C, T, 'listed_with_its_engine_and_connection'),
('unregister-keeps-old-cache', CF, ' self.support.remove(&name);\n self.engines.remove(&name);\n }', ' self.support.remove(&name);\n }', C, T, 'replacing_an_engine_forgets'),
('reader-lost-with-a-new-factory', CC, 'if free.is_some() {', 'if false {', C, T, 'whichever_order'),
('reader-not-given-to-existing-engines', CE, ' kept.set_free_memory(read.clone());\n', '', C, T, 'after_a_model_loaded'),
('replaced-provider-not-retired', CE, ' old.retire();\n', '', C, T, 'changed_mid_job'),
('held-retired-provider-dropped', CE, 'Arc::strong_count(p) > 1 || p.is_busy()', 'p.is_busy()', C, T, 'held_across_a_swap'),
('loading-job-keeps-model-after-swap', CE, ' old.retire();\n', ' old.unload(None);\n', C, T, 'loading_when_its_connection_changes'),
('replaced-provider-hidden', CE, ' merged(out, retiring)\n', ' merged(out, Vec::new())\n', C, T, 'changed_mid_job'),
('engines-judge-loads-apart', CE, 'admission: self.load_admission.clone(),', 'admission: LoadAdmission::default(),', C, T, 'two_engines_at_once'),
('provider-moved-during-listing-listed-twice', CE, 'if !current.iter().any(|(_, _, p)| Arc::ptr_eq(p, &entry.2)) {', 'if true {', C, 'lib', 'listed_once'),
('reader-not-given-to-later-engines', CE, ' kept.set_free_memory(self.free_memory.clone());\n', '', C, T, 'before_its_engines'),
]