User-local lease lifecycle: outage-aware expiry, time limit from the run, idempotent claims, tombstones, failed replies requeue, cool-off per login

The dashboard stamps its liveness every 5 s; job processes expire leases only while
it serves, and for missing heartbeats only after it served the whole idle window.
After a gap every lease's heartbeat, claim and run start move forward by the gap.
A token's drops count at most once in 10 minutes.

The time limit counts from the first running heartbeat; preparing may last 15 min.
A claim repeated with its request id within 5 minutes gets the same reply (kept in
the dashboard's memory; after a restart the orphaned lease goes back unused).
Ended and requeued leases keep their secret hash, so every lease route answers 410.
ok:false requeues with the next attempt and skips that login; a repeated accepted
submit is accepted again. Cool-off is per login, counts drops and 5 failures in a
row within 30 min, and is cleared by login.
This commit is contained in:
aiongg committed 2026-11-19 23:16:13 +00:00
1 parent aaa9fae512
commit 325d7b6529
12 files changed
+382 -142

No files matched your search

+16 -7
View File
@@ -214,17 +214,26 @@ A call goes to a maintainer or operator whose cached GitHub permission is `write
| Event | Rule |
|---|---|
| Claim | The caller's call is leased to the helper; the agent run records the user, harness and the model the helper declared for the tier. The reply carries a lease id and a lease secret. |
| Heartbeat | The helper sends one every 30 seconds with its latest progress line. The server answers 410 when the lease is gone. |
| Expiry | A lease with no heartbeat for 2 minutes, or still running 2 minutes after the call's time limit, ends. The call goes back to the queue with its next attempt, or fails when the prompt's attempts are used up. |
| Release | A helper that cannot run a call (version, platform, missing login, quota refusal) gives the lease back with a reason. The call returns to the queue without using an attempt. |
| Submit | A reply from another model family than claimed is refused and the call is queued again. A failed reply (`ok: false`) fails the call with `helper: <error>`. |
| Claim | The caller's call is leased to the helper; the agent run records the user, harness and the model the helper declared for the tier. The reply carries a lease id and a lease secret. A claim repeated with the same `requestId` within 5 minutes gets the same reply, the same lease and secret included; the replies are kept in the dashboard process's memory only. After a dashboard restart a repeated claim returns the lease it took to the queue without using an attempt, and is answered afresh. |
| Heartbeat | The helper sends one every 30 seconds from the claim on, with its phase (`preparing` or `running`) and latest progress line. The first `running` heartbeat starts the call's time limit. |
| Expiry | A lease ends when it has had no heartbeat for 2 minutes, is still preparing 15 minutes after the claim, or is still running 2 minutes after the call's time limit. The call goes back to the queue with its next attempt, or fails when the prompt's attempts are used up. |
| Dashboard outage | Helpers reach the server only through the dashboard, which stamps every 5 seconds that it serves. A lease ends only while the stamp is fresh (at most 15 seconds old), and for lack of heartbeats only when the dashboard served for the whole 2 minutes. When the dashboard comes back after a gap, every lease's heartbeat, claim and run start move forward by the gap. |
| Release | A helper that cannot prepare or start a call (version, platform, missing login, bundle or checkout errors, quota refusal) gives the lease back with a reason. The call returns to the queue without using an attempt and is not offered to that login for 10 minutes. |
| Submit | A failed reply (`ok: false`, error `helper: <error>`) or a reply from another model family than claimed sends the call back to the queue with its next attempt, not offered to that login for 10 minutes; it fails when the prompt's attempts are used up. A submit repeated after an accepted one is accepted again. |
| Ended lease | Every lease route answers 410 for a secret whose lease was submitted, failed, expired, released or sent back to the queue (the secret hashes are kept 24 hours), and 401 only for a secret never issued; a 410 does not count toward the failed-authentication limit. |
| Caller gone | A queued or leased call whose job stopped polling for 5 minutes fails. |
| Waiting | The runner leaves at most 12 hours of queue time out of a job's deadline and no-progress stop; a call still queued after that is withdrawn ("no helper took the call in time") and the job stops. |
Ended calls are kept 24 hours for task status and follow-ups, then deleted.
**Cool-off.** A token whose leases end by expiry 3 times within 2 hours gets no calls for 6 hours (audit `remote-cooloff`). Its claims return no lease, a reason and the end time; the helper reports `cooled-off`. Lost leases do not count as call failures. An operator clears it with `autocraft remote cooloff clear <token-id|login>` (audit `remote-cooloff-clear`).
**Cool-off.** Cool-off applies to a login, across all its helpers and tokens:
| Cause | Cool-off |
|---|---|
| Leases of the login end by expiry 3 times within 2 hours (each token counts at most once in 10 minutes) | 6 hours |
| The login's helpers send 5 failed replies in a row within 30 minutes (an accepted reply ends the run) | 1 hour |
While it lasts, the login's claims return no lease, a reason and the end time, and its helpers report `cooled-off` (audit `remote-cooloff`). Revoking or replacing a token does not end it. An operator clears it with `autocraft remote cooloff clear <login>` (audit `remote-cooloff-clear`).
### Usage reserve and readiness
@@ -242,7 +251,7 @@ The helper reports its readiness to `POST /api/remote/status` on every change an
| `ready` | The helper claims. |
| `quota-held` | A window is at or below the reserve, its usage could not be read, or the provider refused a run. |
| `paused` | The user ran `pause-until-reset`. |
| `cooled-off` | The server gives the token no calls after repeated lost leases. |
| `cooled-off` | The server gives the login no calls after repeated lost leases or failed replies. |
| `sandbox-failed` | The sandbox check or canary failed. |
| `version-refused` | The server refused the helper's version, or the harness version is not audited (Codex). |
| `harness-missing` | The harness or its login is missing. |
+9 -2
View File
@@ -39,6 +39,7 @@ import { inferenceMode } from "../../remote/mode.ts";
import { HELPERS_PATH, parseStatus, storeStatus } from "../../remote/helpers.ts";
import { ROUTES, type ClaimReply, type StatusReply, type ToolReply } from "../../remote/protocol.ts";
import { userReserve } from "../../remote/reserve.ts";
import { startLiveness } from "../../remote/liveness.ts";
import { archiveOf, callTaskTool, TASK_TOOLS, taskLease, type TaskLease, type ToolResult } from "../../remote/tools.ts";
import { handleDelivery, WEBHOOK_BODY_LIMIT, WEBHOOK_PATH } from "../../github/webhooks.ts";
import { esc, HttpError, parseCookies, plainPage, previewTags, RateLimiter, safeReturnTo } from "./util.ts";
@@ -694,7 +695,7 @@ export function createApp(opts: AppOptions & { routes: Route[] }) {
}
const tool = path.startsWith(`${ROUTES.tool}/`) ? /^([a-z][a-z0-9_-]{0,40})\/([a-z][a-z0-9_]{0,60})$/.exec(path.slice(ROUTES.tool.length + 1)) : null;
if (![ROUTES.bundle, ROUTES.heartbeat, ROUTES.submit, ROUTES.release, ROUTES.tarball].includes(path as never) && !tool) return reply(404, { error: "not found" });
// An unknown secret counts against the address; a known one whose lease ended is answered 410.
// An unknown secret counts against the address; a known one whose lease ended or went back to the queue is answered 410.
if (!remote.leaseState(db, bearer)) {
if (!mcpLimitIp.allow(clientIp(req))) return tooMany();
return reply(401, { error: "a lease secret is required (Authorization: Bearer <secret>)" }, { "www-authenticate": 'Bearer realm="autocraft"' });
@@ -706,7 +707,10 @@ export function createApp(opts: AppOptions & { routes: Route[] }) {
}
if (path === ROUTES.heartbeat) return remote.heartbeat(db, bearer, await readJson(req)) ? reply(200, { ok: true }) : gone();
if (path === ROUTES.release) return remote.releaseCall(db, bearer, await readJson(req)) ? reply(200, { ok: true }) : gone();
if (path === ROUTES.submit) return reply(200, remote.submitCall(db, bearer, await readJson(req, SUBMIT_BODY_LIMIT)));
if (path === ROUTES.submit) {
const out = remote.submitCall(db, bearer, await readJson(req, SUBMIT_BODY_LIMIT));
return out ? reply(200, out) : gone();
}
const lease = taskLease(db, bearer);
if (!lease) return gone();
if (path === ROUTES.tarball) return sendArchive(res, lease, url.searchParams.get("repo") ?? "", url.searchParams.get("sha") ?? "");
@@ -891,6 +895,8 @@ export async function startServer(opts: AppOptions): Promise<{ server: Server; u
const feedbackLimiter = opts.feedbackLimiter ?? new RateLimiter(60, 3600_000);
const app = createApp({ ...opts, feedbackLimiter, routes: opts.routes ?? (await defaultRoutes(feedbackLimiter, opts.githubUserId)) });
const server = createServer({ requestTimeout: 30_000, headersTimeout: 15_000 }, (req, res) => void app.handle(req, res));
// Job processes expire user-local leases only while this process stamps that it serves (src/remote/liveness.ts).
const stopLiveness = startLiveness(opts.db);
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(opts.config.port, opts.config.host, resolve);
@@ -901,6 +907,7 @@ export async function startServer(opts: AppOptions): Promise<{ server: Server; u
server, app,
url: `http://${host}:${addr.port}`,
close: () => new Promise<void>((resolve) => {
stopLiveness();
app.close();
server.closeAllConnections();
server.close(() => resolve());
+4 -2
View File
@@ -180,7 +180,6 @@ const ADDED_COLUMNS: [table: string, column: string, type: string][] = [
["mcp_tokens", "idle_claim_at", "INTEGER"],
["mcp_tokens", "idle_tiers", "TEXT"],
["mcp_tokens", "idle_repos", "TEXT"],
["mcp_tokens", "cooloff_until", "INTEGER"],
// GitHub's type of an item's author (User, Bot, ...); Bot authors are exempt from the outside-author limits.
["items", "author_type", "TEXT"],
// GitHub's numeric id of the author's account; a login string can pass to another account, the id cannot.
@@ -225,6 +224,9 @@ const ADDED_COLUMNS: [table: string, column: string, type: string][] = [
["remote_calls", "released_by", "TEXT COLLATE NOCASE"],
["remote_calls", "released_at", "INTEGER"],
["remote_calls", "root", "TEXT"],
["remote_calls", "started_at", "INTEGER"],
["remote_calls", "claim_id", "TEXT"],
["remote_lease_drops", "login", "TEXT COLLATE NOCASE"],
["dupe_comparisons", "sizes", "TEXT"],
["build_envs", "confidential", "INTEGER NOT NULL DEFAULT 0"],
["build_envs", "confidential_since", "TEXT"],
@@ -260,7 +262,7 @@ const ADDED_INDEXES = [
];
// Columns removed after their first release.
const DROPPED_COLUMNS: [table: string, column: string][] = [["classifications", "topics"], ["classifications", "area"]];
const DROPPED_COLUMNS: [table: string, column: string][] = [["classifications", "topics"], ["classifications", "area"], ["mcp_tokens", "cooloff_until"]];
/** Appends the stored review rows that the history does not hold yet. */
export const HISTORY_COPY_SQL = `INSERT INTO main.opus_review_history (repo, number, head_oid, at, model, mode, verdict, sure, reason, risks, fixes, summary, why, public_review, claims, blocked_by, trigger, job_id)
+126 -55
View File
@@ -16,9 +16,10 @@ import { fillHarness, rootArgs, validRoot } from "../prompts/harness.ts";
import { rootAgents } from "../sandbox/agents.ts";
import { joinParts } from "../prompts/index.ts";
import { scrub } from "../runner/scrub.ts";
import { COOLOFF_REASON, cooloffUntil, noteDrop } from "./cooloff.ts";
import { cooloffOf, noteDrop, noteFailure, noteSuccess } from "./cooloff.ts";
import { dashboardAlive } from "./liveness.ts";
import { acceptedModels, familyOf, isEffort, isTier, meetsEffort, minEfforts, TIERS } from "./mode.ts";
import { PROTOCOL_VERSION, ROUTES, type Claim, type ClaimReply, type HarnessId, type Heartbeat, type Release, type Submit, type SubmitReply, type TaskBundle, type Tier } from "./protocol.ts";
import { CLAIM_ID_TTL_S, PREPARE_MAX_S, PROTOCOL_VERSION, ROUTES, type Claim, type ClaimReply, type HarnessId, type Heartbeat, type Release, type Submit, type SubmitReply, type TaskBundle, type Tier } from "./protocol.ts";
import { claimRepos } from "./my-repos.ts";
import { userReserve } from "./reserve.ts";
import { CALLER_STALE_MS, IDLE_FRESH_MS } from "./room.ts";
@@ -27,9 +28,9 @@ import { PERM_READ_TTL } from "../dash/server/sessions.ts";
import { endChecks, serveChecks } from "./check-relay.ts";
import type { CheckTool } from "../checktool/index.ts";
/** A lease with no heartbeat for this long goes back to the queue. */
/** A lease with no heartbeat for this long, while the dashboard was serving (liveness.ts), goes back to the queue. */
export const LEASE_IDLE_MS = 2 * 60_000;
/** A lease also ends this long after the bundle's own time limit. */
/** A lease also ends this long after the bundle's own time limit, counted from its first `running` heartbeat. */
export const LEASE_MARGIN_MS = 2 * 60_000;
/** How long a follow-up turn waits for the helper that holds its session before the caller falls back to a fresh call. */
export const STICKY_MS = LEASE_IDLE_MS;
@@ -61,11 +62,12 @@ type Row = {
id: number; job_id: number | null; step: string; stage: number; run_id: number | null; prompt_id: string; prompt_version: string;
tier: Tier; effort: string; asked_model: string; bundle: string; state: "queued" | "leased" | "done" | "failed"; items: string; repo: string | null;
cls: number; level: number; since: number; seen_at: number; queued_since: number | null; queued_ms: number;
user_login: string | null; harness: string | null; model: string | null; leased_at: number | null; heartbeat_at: number | null;
user_login: string | null; harness: string | null; model: string | null; leased_at: number | null; heartbeat_at: number | null; started_at: number | null;
token_id: number | null; lease_secret_hash: string | null;
attempt: number; max_attempts: number; timeout_ms: number; parent_id: number | null; sticky_login: string | null; progress: string | null;
reply: string | null; usage: string | null; turns: number | null; session_id: string | null; error: string | null; root: string | null;
};
const COLS = "id, job_id, step, stage, run_id, prompt_id, prompt_version, tier, effort, asked_model, bundle, state, items, repo, class AS cls, level, since, seen_at, queued_since, queued_ms, user_login, harness, model, leased_at, heartbeat_at, attempt, max_attempts, timeout_ms, parent_id, sticky_login, progress, reply, usage, turns, session_id, error, root";
const COLS = "id, job_id, step, stage, run_id, prompt_id, prompt_version, tier, effort, asked_model, bundle, state, items, repo, class AS cls, level, since, seen_at, queued_since, queued_ms, user_login, harness, model, leased_at, heartbeat_at, started_at, token_id, lease_secret_hash, attempt, max_attempts, timeout_ms, parent_id, sticky_login, progress, reply, usage, turns, session_id, error, root";
const rowOf = (db: DatabaseSync, id: number): Row | undefined => db.prepare(`SELECT ${COLS} FROM main.remote_calls WHERE id = ?`).get(id) as Row | undefined;
export type NewCall = {
@@ -104,36 +106,61 @@ export function withdrawJobCalls(db: DatabaseSync, reason: string, jobId?: numbe
return ids.filter((r) => withdrawCall(db, Number(r.id), reason)).length;
}
/** A lost lease goes back to the queue with the next attempt, or fails once the prompt's attempts are used up. */
function requeue(db: DatabaseSync, r: Pick<Row, "id" | "run_id" | "attempt" | "max_attempts" | "leased_at">, reason: string, nowMs: number): boolean {
const fail = r.attempt >= r.max_attempts;
// Compare and set: only the lease that was read ends, so two processes that saw it expire end it once.
const up = db.prepare(`UPDATE main.remote_calls SET state = ?, attempt = attempt + ?, lease_secret_hash = NULL, token_id = NULL, queued_since = ?, ended_at = ?, error = ?, progress = NULL
WHERE id = ? AND state = 'leased' AND attempt = ? AND leased_at IS ?`).run(fail ? "failed" : "queued", fail ? 0 : 1, fail ? null : nowMs, fail ? nowMs : null, reason, r.id, r.attempt, r.leased_at);
if (Number(up.changes) === 1 && r.run_id !== null) finishRun(db, r.run_id, { state: "failed", error: reason, summary: "lease ended" });
return Number(up.changes) === 1;
}
/** Puts a leased call back in the queue without using an attempt (released by its helper, or its token lost the right to run it). */
function unlease(db: DatabaseSync, id: number, runId: number | null, why: string, nowMs: number, releasedBy: string | null = null): void {
const up = db.prepare(`UPDATE main.remote_calls SET state = 'queued', lease_secret_hash = NULL, token_id = NULL, queued_since = ?, error = ?, progress = NULL, released_by = COALESCE(?, released_by), released_at = CASE WHEN ? IS NULL THEN released_at ELSE ? END
WHERE id = ? AND state = 'leased'`).run(nowMs, why, releasedBy, releasedBy, nowMs, id);
if (Number(up.changes) === 1 && runId !== null) finishRun(db, runId, { state: "failed", error: why, summary: "returned to the queue" });
/** Keeps the secret hash of a lease that just ended or went back to the queue, so its routes answer 410 (leaseState); kept as long as ended calls. */
function bury(db: DatabaseSync, id: number, hash: string | null, nowMs: number): void {
if (hash === null) return;
db.prepare("DELETE FROM main.remote_lease_tombstones WHERE at < ?").run(nowMs - KEEP_MS);
db.prepare("INSERT OR IGNORE INTO main.remote_lease_tombstones (secret_hash, call_id, at) VALUES (?, ?, ?)").run(hash, id, nowMs);
}
/**
* The failure backoff of remote calls (registered in src/stages/registry.ts): a lease with no heartbeat for LEASE_IDLE_MS, or past
* its time limit plus LEASE_MARGIN_MS, goes back to the queue (attempt + 1, up to the prompt's attempts); a call whose caller
* stopped polling fails. The lost lease also counts toward its token's flap cool-off. Run by each claim (every call, and old rows are pruned) and by the caller's poll (`id`: that call only).
* A lost or failed lease goes back to the queue with the next attempt, or fails once the prompt's attempts are used up.
* `skipLogin`: the login whose helper failed the call is not offered it again for RELEASE_SKIP_MS.
*/
function requeue(db: DatabaseSync, r: Pick<Row, "id" | "run_id" | "attempt" | "max_attempts" | "leased_at" | "lease_secret_hash">, reason: string, nowMs: number, opts: { skipLogin?: string; summary?: string } = {}): boolean {
const fail = r.attempt >= r.max_attempts;
const skip = opts.skipLogin ?? null;
// Compare and set: only the lease that was read ends, so two processes that saw it expire end it once.
const up = db.prepare(`UPDATE main.remote_calls SET state = ?, attempt = attempt + ?, lease_secret_hash = NULL, token_id = NULL, claim_id = NULL, started_at = NULL, queued_since = ?, ended_at = ?, error = ?, progress = NULL,
released_by = COALESCE(?, released_by), released_at = CASE WHEN ? IS NULL THEN released_at ELSE ? END
WHERE id = ? AND state = 'leased' AND attempt = ? AND leased_at IS ?`).run(fail ? "failed" : "queued", fail ? 0 : 1, fail ? null : nowMs, fail ? nowMs : null, reason, skip, skip, nowMs, r.id, r.attempt, r.leased_at);
if (Number(up.changes) !== 1) return false;
bury(db, r.id, r.lease_secret_hash, nowMs);
if (r.run_id !== null) finishRun(db, r.run_id, { state: "failed", error: reason, summary: opts.summary ?? "lease ended" });
return true;
}
/** Puts a leased call back in the queue without using an attempt (released by its helper, or its token lost the right to run it). */
function unlease(db: DatabaseSync, r: { id: number; runId: number | null; hash: string | null }, why: string, nowMs: number, releasedBy: string | null = null): void {
const up = db.prepare(`UPDATE main.remote_calls SET state = 'queued', lease_secret_hash = NULL, token_id = NULL, claim_id = NULL, started_at = NULL, queued_since = ?, error = ?, progress = NULL, released_by = COALESCE(?, released_by), released_at = CASE WHEN ? IS NULL THEN released_at ELSE ? END
WHERE id = ? AND state = 'leased' AND lease_secret_hash IS ?`).run(nowMs, why, releasedBy, releasedBy, nowMs, r.id, r.hash);
if (Number(up.changes) !== 1) return;
bury(db, r.id, r.hash, nowMs);
if (r.runId !== null) finishRun(db, r.runId, { state: "failed", error: why, summary: "returned to the queue" });
}
/**
* The failure backoff of remote calls (registered in src/stages/registry.ts): a lease with no heartbeat for LEASE_IDLE_MS, a
* lease still preparing PREPARE_MAX_S after the claim, or one running past its time limit plus LEASE_MARGIN_MS goes back to
* the queue (attempt + 1, up to the prompt's attempts); a call whose caller stopped polling fails. Leases end only while the
* dashboard serves, and for lack of heartbeats only when it served for the whole idle window (liveness.ts): helpers reach
* the server only through it. The lost lease counts toward its login's cool-off. Run by each claim (every call, and old
* rows are pruned) and by the caller's poll (`id`: that call only).
*/
export function expireLeases(db: DatabaseSync, nowMs: number = Date.now(), id?: number): void {
const one = id === undefined ? "" : " AND id = ?";
const lost = db.prepare(`SELECT id, run_id, attempt, max_attempts, heartbeat_at, leased_at, timeout_ms, token_id FROM main.remote_calls
WHERE state = 'leased' AND (heartbeat_at < ? OR leased_at + timeout_ms + ? < ?)${one}`).all(nowMs - LEASE_IDLE_MS, LEASE_MARGIN_MS, nowMs, ...(id === undefined ? [] : [id])) as (Row & { timeout_ms: number; token_id: number | null })[];
for (const r of lost) {
const why = Number(r.heartbeat_at) < nowMs - LEASE_IDLE_MS ? `lease expired: no heartbeat for ${LEASE_IDLE_MS / 60_000} min` : "lease expired: past the call's time limit";
// A lost lease counts toward its token's cool-off (cooloff.ts), not as a call failure: the attempt counting stays as it is.
if (requeue(db, r, why, nowMs) && r.token_id !== null) noteDrop(db, Number(r.token_id), nowMs);
const live = dashboardAlive(db, nowMs);
if (live.alive) {
const idle = live.since <= nowMs - LEASE_IDLE_MS ? 1 : 0;
const lost = db.prepare(`SELECT id, run_id, attempt, max_attempts, heartbeat_at, leased_at, started_at, timeout_ms, token_id, user_login, lease_secret_hash FROM main.remote_calls
WHERE state = 'leased' AND ((? AND heartbeat_at < ?) OR (started_at IS NULL AND leased_at + ? < ?) OR (started_at IS NOT NULL AND started_at + timeout_ms + ? < ?))${one}`)
.all(idle, nowMs - LEASE_IDLE_MS, PREPARE_MAX_S * 1000, nowMs, LEASE_MARGIN_MS, nowMs, ...(id === undefined ? [] : [id])) as Row[];
for (const r of lost) {
const why = idle && Number(r.heartbeat_at) < nowMs - LEASE_IDLE_MS ? `lease expired: no heartbeat for ${LEASE_IDLE_MS / 60_000} min`
: r.started_at === null ? `lease expired: still preparing ${PREPARE_MAX_S / 60} min after the claim` : "lease expired: past the call's time limit";
// A lost lease counts toward its login's cool-off (cooloff.ts), and uses an attempt like a failed call.
if (requeue(db, r, why, nowMs) && r.token_id !== null && r.user_login !== null) noteDrop(db, Number(r.token_id), r.user_login, nowMs);
}
}
const gone = db.prepare(`SELECT id FROM main.remote_calls WHERE state IN ('queued', 'leased') AND seen_at < ?${one}`).all(nowMs - CALLER_GONE_MS, ...(id === undefined ? [] : [id])) as { id: number }[];
for (const r of gone) withdrawCall(db, Number(r.id), "the caller stopped waiting", undefined, nowMs);
@@ -157,17 +184,23 @@ export function liveLease(db: DatabaseSync, secret: string | undefined, nowMs: n
WHERE c.id = ? AND c.lease_secret_hash = ? AND c.state = 'leased'`).get(new Date(nowMs).toISOString(), found.id, hash) as
{ id: number; job_id: number | null; run_id: number | null; user_login: string; prompt_id: string; items: string; repo: string | null; root: string | null; ok: string | null } | undefined;
if (r && r.ok === null) {
unlease(db, Number(r.id), r.run_id, "returned to the queue: the token that claimed it may no longer run tasks", nowMs);
unlease(db, { id: Number(r.id), runId: r.run_id, hash }, "returned to the queue: the token that claimed it may no longer run tasks", nowMs);
return null;
}
return r ? { id: Number(r.id), jobId: r.job_id, runId: r.run_id, login: r.user_login, promptId: r.prompt_id, items: JSON.parse(r.items) as ItemRef[], repo: r.repo, root: r.root } : null;
}
/** The state of the call a lease secret was issued for, live or not (for task status); null for an unknown secret. */
export function leaseState(db: DatabaseSync, secret: string | undefined): { id: number; state: Row["state"]; error: string | null } | null {
/**
* The call a lease secret was issued for, whether its lease is live or has ended (submitted, failed, expired, released or
* requeued: the tombstones), or null for a secret never issued. The lease routes answer 410 for an ended lease and 401 only for null.
*/
export function leaseState(db: DatabaseSync, secret: string | undefined): { id: number; live: boolean } | null {
if (!secret || secret.length > 200 || !secret.startsWith(LEASE_PREFIX)) return null;
const r = db.prepare("SELECT id, state, error FROM main.remote_calls WHERE lease_secret_hash = ?").get(sha256hex(secret)) as { id: number; state: Row["state"]; error: string | null } | undefined;
return r ? { id: Number(r.id), state: r.state, error: r.error } : null;
const hash = sha256hex(secret);
const r = db.prepare("SELECT id, state FROM main.remote_calls WHERE lease_secret_hash = ?").get(hash) as { id: number; state: Row["state"] } | undefined;
if (r) return { id: Number(r.id), live: r.state === "leased" };
const t = db.prepare("SELECT call_id FROM main.remote_lease_tombstones WHERE secret_hash = ?").get(hash) as { call_id: number } | undefined;
return t ? { id: Number(t.call_id), live: false } : null;
}
// ---- claims ------------------------------------------------------------------------
@@ -272,9 +305,9 @@ function leaseCall(db: DatabaseSync, id: number, caller: Caller, claim: Claim, m
const secret = `${LEASE_PREFIX}${randomToken(32)}`;
db.exec("BEGIN IMMEDIATE");
try {
const r = db.prepare(`UPDATE main.remote_calls SET state = 'leased', token_id = ?, user_login = ?, harness = ?, harness_version = ?, model = ?, root = ?, lease_secret_hash = ?, leased_at = ?, heartbeat_at = ?,
const r = db.prepare(`UPDATE main.remote_calls SET state = 'leased', token_id = ?, user_login = ?, harness = ?, harness_version = ?, model = ?, root = ?, lease_secret_hash = ?, leased_at = ?, heartbeat_at = ?, started_at = NULL, claim_id = ?,
queued_ms = queued_ms + CASE WHEN queued_since IS NULL THEN 0 ELSE MAX(0, ? - queued_since) END, queued_since = NULL, progress = NULL, error = NULL
WHERE id = ? AND state = 'queued' RETURNING ${COLS}`).get(caller.tokenId, caller.login, claim.harness, claim.harnessVersion, model, claim.root ?? null, sha256hex(secret), nowMs, nowMs, nowMs, id) as Row | undefined;
WHERE id = ? AND state = 'queued' RETURNING ${COLS}`).get(caller.tokenId, caller.login, claim.harness, claim.harnessVersion, model, claim.root ?? null, sha256hex(secret), nowMs, nowMs, claim.requestId, nowMs, id) as Row | undefined;
if (!r) {
db.exec("COMMIT");
return null;
@@ -317,7 +350,7 @@ const memoWrite = (canWrite: (repo: string) => Promise<boolean>): ((repo: string
/** How many queued calls the caller could claim now (same filters and eligibility as claimCall, nothing is leased), for a session's next_task. Follow-ups of a helper's own sessions are not counted. */
export async function claimableCalls(db: DatabaseSync, caller: Caller, opts: { tiers?: readonly Tier[]; models?: Partial<Record<Tier, string>> }, canWrite: (repo: string) => Promise<boolean>, nowMs: number = Date.now()): Promise<number> {
expireLeases(db, nowMs);
if (cooloffUntil(db, caller.tokenId, nowMs) !== null) return 0;
if (cooloffOf(db, caller.login, nowMs) !== null) return 0;
const accepted = acceptedModels(db);
// A tier counts when the model the helper would use for it is of an accepted family; without a model, when the tier accepts any.
const tiers = new Set((opts.tiers ?? TIERS).filter((t) => accepted[t].length > 0 && (opts.models?.[t] === undefined || familyOf(opts.models[t]!, accepted[t]) !== null)));
@@ -335,14 +368,38 @@ export function callStatus(db: DatabaseSync, id: number, login: string): { id: n
}
/**
* Gives the caller the first queued call it may take (claimOrder), filtered by eligibility. `canWrite` reads the caller's live
* write permission on a repo. The reply carries the user's reserve either way.
* Claim replies by agent token and request id, for CLAIM_ID_TTL_S: a repeated claim (its reply was lost) gets the same reply,
* the same lease with the same secret included. Kept in the memory of the dashboard process, the only one that serves
* claims, so the secret is never stored; the promise is kept from the start, so a repeat that arrives while the first is
* still being answered waits for it.
*/
export async function claimCall(db: DatabaseSync, caller: Caller, claim: Claim, canWrite: (repo: string) => Promise<boolean>, nowMs: number = Date.now()): Promise<ClaimReply> {
const claimReplies = new Map<string, { at: number; reply: Promise<ClaimReply> }>();
/**
* Gives the caller the first queued call it may take (claimOrder), filtered by eligibility. `canWrite` reads the caller's live
* write permission on a repo. The reply carries the user's reserve either way. A claim repeated with the same request id
* gets the first one's reply.
*/
export function claimCall(db: DatabaseSync, caller: Caller, claim: Claim, canWrite: (repo: string) => Promise<boolean>, nowMs: number = Date.now()): Promise<ClaimReply> {
for (const [k, v] of claimReplies) if (v.at < nowMs - CLAIM_ID_TTL_S * 1000) claimReplies.delete(k);
const key = `${caller.tokenId}:${claim.requestId}`;
const seen = claimReplies.get(key);
if (seen) return seen.reply;
const reply = claimNew(db, caller, claim, canWrite, nowMs);
claimReplies.set(key, { at: nowMs, reply });
reply.catch(() => { if (claimReplies.get(key)?.reply === reply) claimReplies.delete(key); });
return reply;
}
async function claimNew(db: DatabaseSync, caller: Caller, claim: Claim, canWrite: (repo: string) => Promise<boolean>, nowMs: number): Promise<ClaimReply> {
if (olderVersion(claim.helperVersion, REQUIRED_HELPER)) throw new HttpError(409, `helper ${claim.helperVersion} is older than ${REQUIRED_HELPER}, the oldest this server accepts; download the current helper from ${ROUTES.helper}`);
// A repeat the dashboard no longer remembers (it restarted in between): the lease the first claim took cannot be given
// out again, as only its secret's hash is stored, so it goes back to the queue unused and this claim is answered afresh.
const orphan = db.prepare("SELECT id, run_id, lease_secret_hash FROM main.remote_calls WHERE token_id = ? AND claim_id = ? AND state = 'leased'").get(caller.tokenId, claim.requestId) as { id: number; run_id: number | null; lease_secret_hash: string | null } | undefined;
if (orphan) unlease(db, { id: Number(orphan.id), runId: orphan.run_id, hash: orphan.lease_secret_hash }, "returned to the queue: the claim that leased it was repeated after a server restart", nowMs);
expireLeases(db, nowMs);
const until = cooloffUntil(db, caller.tokenId, nowMs);
if (until !== null) return { lease: null, retryAfter: Math.ceil((until - nowMs) / 1000), reason: COOLOFF_REASON, until, reserve: userReserve(db, caller.login) };
const cooled = cooloffOf(db, caller.login, nowMs);
if (cooled) return { lease: null, retryAfter: Math.ceil((cooled.until - nowMs) / 1000), reason: cooled.reason, until: cooled.until, reserve: userReserve(db, caller.login) };
const accepted = acceptedModels(db), floors = minEfforts(db);
const tiers = new Set((Object.keys(claim.tiers) as Tier[]).filter((t) => familyOf(claim.tiers[t]!, accepted[t]) !== null && meetsEffort(claim.efforts?.[t], floors[t], claim.harness)));
if (!tiers.size) throw new HttpError(400, `no tier of the claim has an accepted model at the minimum effort; accepted: ${(Object.keys(accepted) as Tier[]).map((t) => `${t} ${accepted[t].join("/")}${floors[t] ? ` at effort ${floors[t]} or higher` : ""}`).join(", ")}`);
@@ -392,13 +449,14 @@ export function bundleOf(db: DatabaseSync, secret: string | undefined): TaskBund
/** A plain one-line progress text from the helper: control characters removed, scrubbed, cut. */
const progressLine = (v: unknown): string | null => (typeof v === "string" && v.trim() ? scrub(v.replace(/[\u0000-\u001f\u007f]+/g, " ").trim()).slice(0, PROGRESS_MAX) : null);
/** Records a heartbeat; false when the lease is gone (the route answers 410). */
/** Records a heartbeat; the first `running` one starts the call's time limit. False when the lease is gone (the route answers 410). */
export function heartbeat(db: DatabaseSync, secret: string | undefined, raw: unknown, nowMs: number = Date.now()): boolean {
const l = liveLease(db, secret, nowMs);
if (!l) return false;
const p = (raw as Heartbeat | undefined)?.progress;
const line = progressLine(typeof p === "string" ? unrooted(p, l.root) : p);
db.prepare("UPDATE main.remote_calls SET heartbeat_at = ?, progress = COALESCE(?, progress) WHERE id = ?").run(nowMs, line, l.id);
const h = raw as Partial<Heartbeat> | undefined;
const line = progressLine(typeof h?.progress === "string" ? unrooted(h.progress, l.root) : h?.progress);
db.prepare("UPDATE main.remote_calls SET heartbeat_at = ?, progress = COALESCE(?, progress), started_at = CASE WHEN ? AND started_at IS NULL THEN ? ELSE started_at END WHERE id = ?")
.run(nowMs, line, h?.phase === "running" ? 1 : 0, nowMs, l.id);
if (line && l.runId !== null) setRunProgress(db, l.runId, line);
return true;
}
@@ -416,10 +474,18 @@ const storedTokens = (v: unknown): { input: number; output: number; cacheRead: n
return { input: count(u.input), output: count(u.output), cacheRead: count(u.cacheRead), cacheWrite: count(u.cacheWrite) };
};
/** Takes a helper's result. A reply from another model than the claim declared for the tier is refused and the call goes back to the queue. */
export function submitCall(db: DatabaseSync, secret: string | undefined, raw: unknown, nowMs: number = Date.now()): SubmitReply {
/**
* Takes a helper's result; null when the lease is gone (the route answers 410). A repeat of an accepted reply (the first
* answer was lost) is accepted again. A reply from another model than the claim declared for the tier, and a failed reply
* (`ok: false`), send the call back to the queue with its next attempt, not offered to that login for a while, and count
* toward the login's cool-off; the call fails once its attempts are used up.
*/
export function submitCall(db: DatabaseSync, secret: string | undefined, raw: unknown, nowMs: number = Date.now()): SubmitReply | null {
const l = liveLease(db, secret, nowMs);
if (!l) return { accepted: false, reason: "the lease is gone (expired, withdrawn or already submitted)" };
if (!l) {
const done = secret?.startsWith(LEASE_PREFIX) && db.prepare("SELECT 1 FROM main.remote_calls WHERE lease_secret_hash = ? AND state = 'done'").get(sha256hex(secret));
return done ? { accepted: true } : null;
}
if (!raw || typeof raw !== "object" || Array.isArray(raw)) throw new HttpError(400, "the body must be a Submit object");
const s = raw as Submit;
const usage = usageOf((s as { usage?: unknown }).usage);
@@ -431,7 +497,7 @@ export function submitCall(db: DatabaseSync, secret: string | undefined, raw: un
const families = acceptedModels(db)[r.tier];
const family = familyOf(r.model ?? "", families);
if (family === null || familyOf(s.model, families) !== family) {
requeue(db, r, `the harness reported model ${s.model.slice(0, 100)}, not the ${family ?? r.model} family claimed for the tier`, nowMs);
if (requeue(db, r, `the harness reported model ${s.model.slice(0, 100)}, not the ${family ?? r.model} family claimed for the tier`, nowMs, { skipLogin: l.login, summary: "refused" })) noteFailure(db, l.login, nowMs);
return { accepted: false, reason: `the claim declared ${r.model} for this tier` };
}
const byModel: Record<string, ReturnType<typeof usageOf>> = {};
@@ -442,6 +508,7 @@ export function submitCall(db: DatabaseSync, secret: string | undefined, raw: un
const costUsd = typeof s.costUsd === "number" && Number.isFinite(s.costUsd) && s.costUsd >= 0 ? s.costUsd : null;
db.prepare("UPDATE main.remote_calls SET state = 'done', reply = ?, usage = ?, turns = ?, session_id = ?, duration_ms = ?, ended_at = ?, progress = NULL WHERE id = ? AND state = 'leased'")
.run(unrooted(s.text, r.root), JSON.stringify({ ...(usage ?? usageOf({})), ...(Object.keys(byModel).length > 0 && { byModel }), ...(costUsd !== null && { costUsd }) }), turns, sessionId, duration, nowMs, l.id);
noteSuccess(db, l.login);
// The claim named the model the helper meant to run (often an alias); the reply names the one that ran.
if (r.run_id !== null) {
db.prepare("UPDATE main.agent_runs SET reported_model = ? WHERE id = ?").run(s.model.slice(0, 100), r.run_id);
@@ -450,17 +517,21 @@ export function submitCall(db: DatabaseSync, secret: string | undefined, raw: un
return { accepted: true };
}
if (s.ok !== false || typeof s.error !== "string") throw new HttpError(400, "ok must be true with text and model, or false with error");
db.prepare("UPDATE main.remote_calls SET state = 'failed', error = ?, usage = ?, duration_ms = ?, ended_at = ?, progress = NULL WHERE id = ? AND state = 'leased'")
.run(scrubError(`helper: ${unrooted(s.error, l.root)}`), usage ? JSON.stringify(usage) : null, duration, nowMs, l.id);
const r = rowOf(db, l.id)!;
if (requeue(db, r, scrubError(`helper: ${unrooted(s.error, l.root)}`), nowMs, { skipLogin: l.login, summary: "helper failed" })) {
noteFailure(db, l.login, nowMs);
// The last attempt's tokens stay with the failed call.
db.prepare("UPDATE main.remote_calls SET usage = ?, duration_ms = ? WHERE id = ? AND state = 'failed'").run(usage ? JSON.stringify(usage) : null, duration, l.id);
}
return { accepted: true };
}
/** The helper gives the lease back (too old, or its platform or harness login is missing): the call returns to the queue without using an attempt. False when the lease is gone. */
/** The helper gives the lease back (it could not prepare or start the run): the call returns to the queue without using an attempt and is not offered to that login for a while. False when the lease is gone. */
export function releaseCall(db: DatabaseSync, secret: string | undefined, raw: unknown, nowMs: number = Date.now()): boolean {
const l = liveLease(db, secret, nowMs);
if (!l) return false;
const reason = progressLine((raw as Release | undefined)?.reason) ?? "released by the helper";
unlease(db, l.id, l.runId, `released by the helper: ${reason}`, nowMs, l.login);
unlease(db, { id: l.id, runId: l.runId, hash: sha256hex(secret!) }, `released by the helper: ${reason}`, nowMs, l.login);
return true;
}
+4 -5
View File
@@ -1,4 +1,4 @@
// `autocraft inference [server|user-local]`: shows or sets the inference mode (src/remote/mode.ts). `autocraft remote cooloff clear`: ends a helper's lease cool-off (src/remote/cooloff.ts).
// `autocraft inference [server|user-local]`: shows or sets the inference mode (src/remote/mode.ts). `autocraft remote cooloff clear <login>`: ends the cool-off of a login's helpers (src/remote/cooloff.ts).
// `autocraft remote token <login>`: makes a running agent token for a maintainer or operator and prints only the token, so it
// can be piped into `autocraft-task login` on the same host without anyone seeing it.
import { connect } from "../db.ts";
@@ -25,15 +25,14 @@ export const commands = {
},
},
remote: {
summary: "remote cooloff clear <token-id|login>: lets a helper claim calls again after its lease cool-off; remote token <login> [--label <text>]: a running agent token on stdout, for piping into autocraft-task login",
summary: "remote cooloff clear <login>: lets the login's helpers claim calls again after a cool-off; remote token <login> [--label <text>]: a running agent token on stdout, for piping into autocraft-task login",
run(args: string[]): void {
if (args[0] === "token") return makeToken(args.slice(1));
if (args[0] !== "cooloff" || args[1] !== "clear" || !args[2] || args.length > 3) {
console.error("usage: autocraft remote cooloff clear <token-id|login>");
console.error("usage: autocraft remote cooloff clear <login>");
process.exit(1);
}
const n = clearCooloff(connect(), args[2], "cli");
console.log(n ? `cleared the cool-off of ${n} token${n === 1 ? "" : "s"}` : `no token of ${args[2]} is cooling off`);
console.log(clearCooloff(connect(), args[2], "cli") ? `cleared the cool-off of ${args[2]}` : `${args[2]} is not cooling off`);
},
},
};
+54 -32
View File
@@ -1,45 +1,67 @@
// Flap cool-off of the user-local mode: a token whose leases keep ending without a release or a reply (the helper lost its
// heartbeat, or ran past the call's time limit) gets no calls for a while, so a helper that keeps dropping them stops
// holding calls other helpers could run. Drops are not call failures: the call's attempt counting is unchanged (calls.ts).
// Cool-off of the user-local mode, per login: a login whose helpers keep losing leases (no heartbeat, past a time limit) or
// keep failing calls (`ok: false` replies in a row) gets no calls for a while, so a broken helper stops holding and failing
// calls other helpers could run. Revoking a token does not end it; an operator does (`autocraft remote cooloff clear`).
import type { DatabaseSync } from "node:sqlite";
import { audit } from "../dash/audit.ts";
/** Lease drops of one token within DROP_WINDOW_MS that start a cool-off. */
/** Lease drops of one login within DROP_WINDOW_MS that start a cool-off. */
export const DROP_LIMIT = 3;
export const DROP_WINDOW_MS = 2 * 3600_000;
/** How long a cooled-off token gets nothing. */
/** A token's drops count at most once in this long: one outage or one stuck machine is one drop. */
export const DROP_DEDUPE_MS = 10 * 60_000;
/** How long lost leases cool a login off. */
export const COOLOFF_MS = 6 * 3600_000;
/** Failed replies in a row, within FAILURE_WINDOW_MS of the first, that start a cool-off. */
export const FAILURE_LIMIT = 5;
export const FAILURE_WINDOW_MS = 30 * 60_000;
/** How long failed replies cool a login off. */
export const FAILURE_COOLOFF_MS = 3600_000;
/** Records a dropped lease of `tokenId`; starts the cool-off (and counts afresh afterwards) when it is the DROP_LIMIT-th within the window. */
export function noteDrop(db: DatabaseSync, tokenId: number, nowMs: number): void {
db.prepare("DELETE FROM main.remote_lease_drops WHERE token_id = ? AND at < ?").run(tokenId, nowMs - DROP_WINDOW_MS);
db.prepare("INSERT INTO main.remote_lease_drops (token_id, at) VALUES (?, ?)").run(tokenId, nowMs);
const n = Number((db.prepare("SELECT COUNT(*) AS n FROM main.remote_lease_drops WHERE token_id = ?").get(tokenId) as { n: number }).n);
if (n < DROP_LIMIT) return;
const t = db.prepare("SELECT login FROM main.mcp_tokens WHERE id = ?").get(tokenId) as { login: string } | undefined;
if (!t) return;
db.prepare("UPDATE main.mcp_tokens SET cooloff_until = ? WHERE id = ?").run(nowMs + COOLOFF_MS, tokenId);
db.prepare("DELETE FROM main.remote_lease_drops WHERE token_id = ?").run(tokenId);
audit(db, t.login, "remote-cooloff", String(tokenId), { until: nowMs + COOLOFF_MS });
export const DROP_REASON = `its leases were lost ${DROP_LIMIT} times within ${DROP_WINDOW_MS / 3600_000} hours (no heartbeat, or past the call's time limit)`;
export const FAILURE_REASON = `its helpers failed ${FAILURE_LIMIT} calls in a row within ${FAILURE_WINDOW_MS / 60_000} minutes`;
function coolOff(db: DatabaseSync, login: string, until: number, reason: string): void {
db.prepare(`INSERT INTO main.remote_cooloffs (login, until, reason, failures, failures_since) VALUES (?, ?, ?, 0, NULL)
ON CONFLICT (login) DO UPDATE SET until = excluded.until, reason = excluded.reason, failures = 0, failures_since = NULL`).run(login, until, reason);
db.prepare("DELETE FROM main.remote_lease_drops WHERE login = ?").run(login);
audit(db, login, "remote-cooloff", login, { until, reason });
}
/** When the token may claim again, or null when it is not cooling off. */
export function cooloffUntil(db: DatabaseSync, tokenId: number, nowMs: number): number | null {
const r = db.prepare("SELECT cooloff_until FROM main.mcp_tokens WHERE id = ?").get(tokenId) as { cooloff_until: number | null } | undefined;
return r?.cooloff_until != null && r.cooloff_until > nowMs ? Number(r.cooloff_until) : null;
/** Records a lost lease of `tokenId`, held by `login`; the DROP_LIMIT-th drop of the login within the window starts the cool-off. */
export function noteDrop(db: DatabaseSync, tokenId: number, login: string, nowMs: number): void {
if (db.prepare("SELECT 1 FROM main.remote_lease_drops WHERE token_id = ? AND at > ?").get(tokenId, nowMs - DROP_DEDUPE_MS)) return;
db.prepare("DELETE FROM main.remote_lease_drops WHERE at < ?").run(nowMs - DROP_WINDOW_MS);
db.prepare("INSERT INTO main.remote_lease_drops (token_id, login, at) VALUES (?, ?, ?)").run(tokenId, login, nowMs);
const n = Number((db.prepare("SELECT COUNT(*) AS n FROM main.remote_lease_drops WHERE login = ?").get(login) as { n: number }).n);
if (n >= DROP_LIMIT) coolOff(db, login, nowMs + COOLOFF_MS, DROP_REASON);
}
export const COOLOFF_REASON = `its leases were lost ${DROP_LIMIT} times within ${DROP_WINDOW_MS / 3600_000} hours (no heartbeat, or past the call's time limit)`;
/** Records a failed reply of `login`'s helper; the FAILURE_LIMIT-th in a row within the window starts the cool-off. */
export function noteFailure(db: DatabaseSync, login: string, nowMs: number): void {
const r = db.prepare("SELECT failures, failures_since FROM main.remote_cooloffs WHERE login = ?").get(login) as { failures: number; failures_since: number | null } | undefined;
const fresh = !r || r.failures_since === null || Number(r.failures_since) < nowMs - FAILURE_WINDOW_MS;
const failures = fresh ? 1 : Number(r.failures) + 1;
if (failures >= FAILURE_LIMIT) return coolOff(db, login, nowMs + FAILURE_COOLOFF_MS, FAILURE_REASON);
db.prepare(`INSERT INTO main.remote_cooloffs (login, failures, failures_since) VALUES (?, ?, ?)
ON CONFLICT (login) DO UPDATE SET failures = excluded.failures, failures_since = excluded.failures_since`).run(login, failures, fresh ? nowMs : r!.failures_since);
}
/** Clears the cool-off of the tokens matching an id or a login; returns how many were cooling off. */
export function clearCooloff(db: DatabaseSync, who: string, by: string, nowMs: number = Date.now()): number {
const rows = (/^\d+$/.test(who)
? db.prepare("SELECT id, login FROM main.mcp_tokens WHERE id = ? AND cooloff_until > ?").all(Number(who), nowMs)
: db.prepare("SELECT id, login FROM main.mcp_tokens WHERE login = ? COLLATE NOCASE AND cooloff_until > ?").all(who, nowMs)) as { id: number; login: string }[];
for (const r of rows) {
db.prepare("UPDATE main.mcp_tokens SET cooloff_until = NULL WHERE id = ?").run(r.id);
db.prepare("DELETE FROM main.remote_lease_drops WHERE token_id = ?").run(r.id);
audit(db, by, "remote-cooloff-clear", String(r.id), { login: r.login });
}
return rows.length;
/** An accepted reply ends the login's run of failures. */
export function noteSuccess(db: DatabaseSync, login: string): void {
db.prepare("UPDATE main.remote_cooloffs SET failures = 0, failures_since = NULL WHERE login = ? AND failures > 0").run(login);
}
/** When the login's helpers may claim again and why they may not now, or null when the login is not cooling off. */
export function cooloffOf(db: DatabaseSync, login: string, nowMs: number): { until: number; reason: string } | null {
const r = db.prepare("SELECT until, reason FROM main.remote_cooloffs WHERE login = ?").get(login) as { until: number | null; reason: string | null } | undefined;
return r?.until != null && Number(r.until) > nowMs ? { until: Number(r.until), reason: r.reason ?? DROP_REASON } : null;
}
/** Ends the login's cool-off and forgets its drops and failures; false when it was not cooling off. */
export function clearCooloff(db: DatabaseSync, login: string, by: string, nowMs: number = Date.now()): boolean {
const was = cooloffOf(db, login, nowMs) !== null;
db.prepare("DELETE FROM main.remote_cooloffs WHERE login = ?").run(login);
db.prepare("DELETE FROM main.remote_lease_drops WHERE login = ?").run(login);
if (was) audit(db, by, "remote-cooloff-clear", login, {});
return was;
}
+7 -6
View File
@@ -3,7 +3,7 @@
// last status is `ready` and fresh (room.ts); the Connect page and my_work list the user's own.
import type { DatabaseSync } from "node:sqlite";
import { HttpError } from "../dash/server/util.ts";
import { COOLOFF_REASON } from "./cooloff.ts";
import { cooloffOf } from "./cooloff.ts";
import { isTier } from "./mode.ts";
import { READINESS, STATUS_EVERY_S, type HarnessId, type HelperInfo, type HelperStatus, type Readiness, type Reserve, type Tier, type UsageReading, type UsageWindow } from "./protocol.ts";
@@ -78,19 +78,20 @@ export function storeStatus(db: DatabaseSync, tokenId: number, login: string, s:
/** The login's helpers: every live token that may run tasks, with its last status. */
export function listHelpers(db: DatabaseSync, login: string, nowMs: number = Date.now()): HelperInfo[] {
const rows = db.prepare(`SELECT t.id, t.label, t.last_used_at, t.cooloff_until, h.readiness, h.until, h.reason, h.harness, h.models, h.reserve, h.usage, h.reported_at
const rows = db.prepare(`SELECT t.id, t.label, t.last_used_at, h.readiness, h.until, h.reason, h.harness, h.models, h.reserve, h.usage, h.reported_at
FROM main.mcp_tokens t LEFT JOIN main.remote_helpers h ON h.token_id = t.id
WHERE t.login = ? AND t.can_run = 1 AND t.expires_at > ? ORDER BY t.created_at DESC, t.id DESC`).all(login, new Date(nowMs).toISOString()) as
{ id: number; label: string | null; last_used_at: string | null; cooloff_until: number | null; readiness: Readiness | null; until: string | null; reason: string | null; harness: HarnessId | null; models: string | null; reserve: string | null; usage: string | null; reported_at: number | null }[];
{ id: number; label: string | null; last_used_at: string | null; readiness: Readiness | null; until: string | null; reason: string | null; harness: HarnessId | null; models: string | null; reserve: string | null; usage: string | null; reported_at: number | null }[];
const cooloff = cooloffOf(db, login, nowMs);
return rows.map((r) => {
const reported = r.reported_at === null ? null : Number(r.reported_at);
const used = r.last_used_at ? Date.parse(r.last_used_at) : null;
const seen = Math.max(reported ?? -Infinity, used ?? -Infinity);
// The server's own cool-off of the token shows even before the helper has reported it.
const cooled = r.cooloff_until !== null && Number(r.cooloff_until) > nowMs && r.readiness !== "cooled-off";
// The server's own cool-off of the login shows even before the helper has reported it.
const cooled = cooloff !== null && r.readiness !== "cooled-off";
return {
tokenId: Number(r.id), label: r.label, slots: null, busy: null, lastError: null,
...(cooled ? { readiness: "cooled-off" as const, until: new Date(Number(r.cooloff_until)).toISOString(), reason: COOLOFF_REASON } : { readiness: r.readiness, until: r.until, reason: r.reason }),
...(cooled ? { readiness: "cooled-off" as const, until: new Date(cooloff.until).toISOString(), reason: cooloff.reason } : { readiness: r.readiness, until: r.until, reason: r.reason }),
fresh: reported !== null && nowMs - reported <= STATUS_FRESH_MS,
lastSeen: Number.isFinite(seen) ? new Date(seen).toISOString() : null,
harness: r.harness, models: r.models ? JSON.parse(r.models) as Partial<Record<Tier, string>> : {},
+55
View File
@@ -0,0 +1,55 @@
// The dashboard's liveness stamp. Helpers reach the server only through the dashboard, so while it is down, stalled or
// restarting no heartbeat can arrive, and a lease must not be counted as lost for that. The dashboard stamps the
// `remote_liveness` row every ALIVE_EVERY_MS; job processes, which expire leases (calls.ts expireLeases), read it.
import type { DatabaseSync } from "node:sqlite";
export const ALIVE_EVERY_MS = 5_000;
/** A stamp older than this means the dashboard is not serving: down, stalled or restarting. */
export const ALIVE_STALE_MS = 3 * ALIVE_EVERY_MS;
/**
* Writes the stamp. After a gap (the previous stamp is stale) every leased call's heartbeat, lease and run start move
* forward by the gap, capped at now, so neither the idle window nor the time limit counts the time no helper could reach
* the server; the unbroken run of stamps then starts afresh.
*/
export function stampAlive(db: DatabaseSync, nowMs: number = Date.now()): void {
// Another request's transaction is open on this connection (across an await): the next tick stamps.
if (db.isTransaction) return;
db.exec("BEGIN IMMEDIATE");
try {
const r = db.prepare("SELECT alive_at FROM main.remote_liveness WHERE id = 1").get() as { alive_at: number } | undefined;
if (!r) db.prepare("INSERT INTO main.remote_liveness (id, alive_at, since) VALUES (1, ?, ?)").run(nowMs, nowMs);
else if (nowMs - Number(r.alive_at) > ALIVE_STALE_MS) {
const gap = nowMs - Number(r.alive_at);
db.prepare(`UPDATE main.remote_calls SET heartbeat_at = MIN(heartbeat_at + ?1, ?2), leased_at = MIN(leased_at + ?1, ?2),
started_at = CASE WHEN started_at IS NULL THEN NULL ELSE MIN(started_at + ?1, ?2) END
WHERE state = 'leased'`).run(gap, nowMs);
db.prepare("UPDATE main.remote_liveness SET alive_at = ?, since = ? WHERE id = 1").run(nowMs, nowMs);
} else db.prepare("UPDATE main.remote_liveness SET alive_at = ? WHERE id = 1").run(nowMs);
db.exec("COMMIT");
} catch (e) {
try { db.exec("ROLLBACK"); } catch { /* already ended */ }
throw e;
}
}
/**
* Whether the dashboard is serving now, and since when without a gap. A database the dashboard never stamped (no
* dashboard has run against it) counts as always alive.
*/
export function dashboardAlive(db: DatabaseSync, nowMs: number): { alive: boolean; since: number } {
const r = db.prepare("SELECT alive_at, since FROM main.remote_liveness WHERE id = 1").get() as { alive_at: number; since: number } | undefined;
if (!r) return { alive: true, since: -Infinity };
return { alive: nowMs - Number(r.alive_at) <= ALIVE_STALE_MS, since: Number(r.since) };
}
/** Starts the stamp timer in the dashboard process; returns the function that stops it. */
export function startLiveness(db: DatabaseSync): () => void {
const tick = (): void => {
try { stampAlive(db); } catch (e) { console.log(`liveness stamp not written: ${(e as Error).message}`); }
};
tick();
const t = setInterval(tick, ALIVE_EVERY_MS);
t.unref();
return () => clearInterval(t);
}
+28 -4
View File
@@ -240,14 +240,36 @@ CREATE TABLE IF NOT EXISTS mcp_tokens (
created_at TEXT NOT NULL, last_used_at TEXT, expires_at TEXT NOT NULL,
can_run INTEGER NOT NULL DEFAULT 0, -- the token's helper may claim model calls in the user-local inference mode
idle_claim_at INTEGER, -- ms since epoch: the token's last claim that found nothing to take (src/remote/room.ts)
idle_tiers TEXT, idle_repos TEXT, -- JSON: the tiers and repos (null: any) that empty claim asked for
cooloff_until INTEGER -- ms since epoch: the token gets no calls until then, after repeated lost leases (src/remote/cooloff.ts)
idle_tiers TEXT, idle_repos TEXT -- JSON: the tiers and repos (null: any) that empty claim asked for
);
-- Each lease of a token that ended without a release or a reply (lost heartbeat, past its time limit), newest 2 hours kept.
-- Leases that ended without a release or a reply (lost heartbeat, past a time limit), at most one per token in 10 minutes,
-- newest 2 hours kept; counted per login toward its cool-off (src/remote/cooloff.ts).
CREATE TABLE IF NOT EXISTS remote_lease_drops (
token_id INTEGER NOT NULL, at INTEGER NOT NULL
token_id INTEGER NOT NULL, at INTEGER NOT NULL,
login TEXT COLLATE NOCASE
);
CREATE INDEX IF NOT EXISTS remote_lease_drops_token ON remote_lease_drops(token_id, at);
-- The cool-off of a login's helpers in the user-local inference mode (src/remote/cooloff.ts): no calls until `until`.
CREATE TABLE IF NOT EXISTS remote_cooloffs (
login TEXT PRIMARY KEY COLLATE NOCASE,
until INTEGER, -- ms since epoch; null or past when not cooling off
reason TEXT, -- why, as the claim reply and the Connect page show it
failures INTEGER NOT NULL DEFAULT 0, -- failed replies (ok: false) in a row
failures_since INTEGER -- the first of them
);
-- The secret hashes of leases that ended or went back to the queue, so their routes answer 410 rather than 401 (kept 24 hours).
CREATE TABLE IF NOT EXISTS remote_lease_tombstones (
secret_hash TEXT PRIMARY KEY,
call_id INTEGER NOT NULL,
at INTEGER NOT NULL -- ms since epoch
);
-- The dashboard's liveness stamp (src/remote/liveness.ts), one row: written every 5 seconds; `since` is when the current
-- unbroken run of stamps began. Leases expire only while the dashboard was alive for the whole idle window.
CREATE TABLE IF NOT EXISTS remote_liveness (
id INTEGER PRIMARY KEY CHECK (id = 1),
alive_at INTEGER NOT NULL,
since INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS mcp_tokens_login ON mcp_tokens(login);
-- The last status of each helper (an agent token that may run tasks) in the user-local inference mode (src/remote/helpers.ts):
-- readiness (src/remote/protocol.ts Readiness), until and reason as the helper sent them, JSON of its models, reserve and usage.
@@ -760,6 +782,8 @@ CREATE TABLE IF NOT EXISTS remote_calls (
lease_secret_hash TEXT, -- sha256 of the lease secret
token_id INTEGER, -- the agent token that claimed the lease; the lease ends when it may no longer run tasks
leased_at INTEGER, heartbeat_at INTEGER,
started_at INTEGER, -- the lease's first `running` heartbeat: the call's time limit counts from here
claim_id TEXT, -- the claim's request id (protocol Claim.requestId) that leased it
attempt INTEGER NOT NULL DEFAULT 1,
max_attempts INTEGER NOT NULL, -- the prompt's attempts limit
timeout_ms INTEGER NOT NULL, -- the bundle's wall-clock limit
+2 -2
View File
@@ -496,8 +496,8 @@ export const GLOBAL_WAITS: { name: string; why: string; howLong: string; showsAs
{ name: "Security fix build", why: "The private fix of a security finding waits for its build tasks in a confidential build environment; no job holds the wait", howLong: "Until the tasks end, at most 4 hours from when a confidential environment was ready for them; then they are cancelled and the round is read as failed", showsAs: "\"Waiting for a confidential build environment.\" on the task while none is ready; the confidential-env health check", kind: "external" },
{ name: "Security scan test", why: "The scan test of a finding (feature Run scan tests) waits for its build task in a confidential build environment; no job holds the wait", howLong: "Until the task ends, at most 4 hours from when a confidential environment was ready for it; then it is cancelled and the finding reads as could-not-build", showsAs: "Test badge on the finding; \"Waiting for a confidential build environment.\" on the task while none is ready; the confidential-env health check", kind: "external" },
{ name: "User-local helpers", why: "In the user-local inference mode a Claude model call waits in the remote call queue until the helper of a maintainer or operator who may take it claims it (in the model queue's order). A follow-up turn waits only for the helper that holds its session, at most 2 minutes; then a resumed session is asked anew as a fresh call, while a lint fix turn or a clippy reminder is skipped", howLong: "Until a helper claims it. Up to 12 hours of a job's queue time (the time in which any of its calls waited) is left out of its deadline and no-progress stop; past that the job is stopped as past its deadline, so a call no eligible helper takes does not hold the job and its item", showsAs: "\"waiting for a user-local helper\" in the job log", kind: "external" },
{ name: "Lost remote lease", why: "A helper that sends no heartbeat for 2 minutes, or runs past the call's time limit plus 2 minutes, loses the call: it goes back to the queue for any helper; checked on the caller's poll and on every claim, no timer", howLong: "At once, up to the prompt's attempts; then the call fails as a local call does", showsAs: "The agent run, failed with \"lease expired\"", kind: "backoff" },
{ name: "Remote helper cool-off", why: "A helper token whose leases were lost 3 times within 2 hours (no heartbeat, or past the call's time limit) is offered no calls, so a helper that keeps dropping them stops holding calls other helpers could run. Dropped leases do not count as call failures", howLong: "6 hours from the third loss, or until an operator runs `autocraft remote cooloff clear <token-id|login>`", showsAs: "The helper's claim reply names the reason and the time; it prints them once and waits", kind: "backoff" },
{ name: "Lost remote lease", why: "A helper that sends no heartbeat for 2 minutes while the dashboard serves, that is still preparing 15 minutes after the claim, or that runs past the call's time limit (counted from its first running heartbeat) plus 2 minutes loses the call: it goes back to the queue for any helper. A failed reply sends it back the same way, not offered to that user's helpers for 10 minutes. Time in which the dashboard was down or stalled does not count. Checked on the caller's poll and on every claim, no timer", howLong: "At once, up to the prompt's attempts; then the call fails as a local call does", showsAs: "The agent run, failed with \"lease expired\" or the helper's error", kind: "backoff" },
{ name: "Remote helper cool-off", why: "A user whose helpers lost leases 3 times within 2 hours (at most one per token in 10 minutes), or failed 5 calls in a row within 30 minutes, is offered no calls, so a broken helper stops holding and failing calls other helpers could run. Revoking a token does not end it", howLong: "6 hours from the third loss, 1 hour from the fifth failure, or until an operator runs `autocraft remote cooloff clear <login>`", showsAs: "The helper's claim reply names the reason and the time; it prints them once and waits", kind: "backoff" },
{ name: "Claude usage pause", why: "Account headroom", howLong: "Until usage reading recovers; re-read every minute", showsAs: "Claude pause reason", kind: "external" },
{ name: "Rate limits", why: "GitHub. Writes are also paced under GitHub's limits: at most 60 a minute and 400 an hour per identity and repo owner (one key for the box account, which runs every legacy repo, operator presses on those included; only `user:` identities, an operator's own token, are exempt), and a key pauses 60 seconds after a secondary-limit failure. A write waits at most 15 seconds for its turn, then fails as paced and the planner retries it", howLong: "Per the API's reset; write pacing frees as the minute and hour windows move", showsAs: "Job log", kind: "external" },
{ name: "Planner fallback", why: "Safety net; the planner normally runs on a change", howLong: "5 minutes", showsAs: "none", kind: "backoff" },
+76 -26
View File
@@ -13,7 +13,8 @@ import { connect } from "../src/db.ts";
import { RateLimiter } from "../src/dash/server/util.ts";
import type { User } from "../src/dash/types.ts";
import { SAMPLE_OPUS_PRS } from "../src/prompts/samples.ts";
import { bundleOf, claimableCalls, claimCall, CLAIM_RETRY_S, expireLeases, heartbeat, withdrawCall, submitCall, type Caller } from "../src/remote/calls.ts";
import { bundleOf, claimableCalls, claimCall, CLAIM_RETRY_S, expireLeases, heartbeat, leaseState, withdrawCall, submitCall, type Caller } from "../src/remote/calls.ts";
import { stampAlive } from "../src/remote/liveness.ts";
import { clearCooloff } from "../src/remote/cooloff.ts";
import * as userSettings from "../src/dash/api/user-settings.ts";
import type { Req } from "../src/dash/types.ts";
@@ -38,12 +39,14 @@ describe("user-local remote calls", () => {
db.prepare("INSERT INTO access (login, role) VALUES (?, 'maintainer')").run(login);
callers[login] = { tokenId: createToken(db, login, "helper", false, true).row.id, login, role: "maintainer" };
}
const claim: Claim = { requestId: "test-claim-0001", protocol: PROTOCOL_VERSION, helperVersion: "1.1.0", harness: "claude", harnessVersion: "2.1", tiers: { top: "opus" } };
let claims = 0;
/** A claim body with its own request id: a repeated id gets the first claim's reply. */
const claim = (): Claim => ({ requestId: `test-claim-${String(++claims).padStart(4, "0")}`, protocol: PROTOCOL_VERSION, helperVersion: "1.1.0", harness: "claude", harnessVersion: "2.1", tiers: { top: "opus" } });
const canWrite = async () => true;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
const leaseFor = async (login: string) => {
for (let i = 0; i < 200; i++) {
const r = await claimCall(db, callers[login]!, claim, canWrite);
const r = await claimCall(db, callers[login]!, claim(), canWrite);
if (r.lease) return r.lease;
if (db.prepare("SELECT 1 FROM remote_calls WHERE state = 'queued'").get()) return null;
await sleep(5);
@@ -65,7 +68,7 @@ describe("user-local remote calls", () => {
assert.equal(await leaseFor("alice"), null, "the PR's author");
assert.equal(await leaseFor("carol"), null, "a committer on the PR");
assert.equal(await leaseFor("dave"), null, "a co-author of a commit");
await assert.rejects(claimCall(db, callers.bob!, { ...claim, helperVersion: "0.9.0" }, canWrite), /older than/);
await assert.rejects(claimCall(db, callers.bob!, { ...claim(), helperVersion: "0.9.0" }, canWrite), /older than/);
const lease = (await leaseFor("bob"))!;
assert.match(lease.title, /^review\.opus on photocraft#1287/);
const b = bundleOf(db, lease.secret)!;
@@ -105,23 +108,27 @@ describe("user-local remote calls", () => {
const first = (await leaseFor("bob"))!;
db.prepare("UPDATE remote_calls SET heartbeat_at = heartbeat_at - 3 * 60000 WHERE id = ?").run(first.id);
let second = null;
for (let i = 0; i < 200 && !second; i++) { second = (await claimCall(db, callers.bob!, claim, canWrite)).lease; if (!second) await sleep(5); }
for (let i = 0; i < 200 && !second; i++) { second = (await claimCall(db, callers.bob!, claim(), canWrite)).lease; if (!second) await sleep(5); }
assert.equal(second!.id, first.id, "the same call, leased again");
assert.equal(heartbeat(db, first.secret, {}), false);
assert.equal(done(first.secret, "stale").accepted, false);
assert.equal(done(first.secret, "stale"), null, "the route answers 410");
assert.deepEqual(leaseState(db, first.secret), { id: first.id, live: false }, "a requeued lease's secret is known: 410, not 401");
assert.equal(leaseState(db, "aclease_never-issued"), null);
assert.equal((db.prepare("SELECT attempt FROM remote_calls WHERE id = ?").get(first.id) as { attempt: number }).attempt, 2);
// A reply from another model family than the claim declared is refused.
assert.equal(submitCall(db, second!.secret, { ok: true, text: "x", model: "claude-sonnet-4-5", usage: { inputTokens: 1, outputTokens: 1 }, durationMs: 1 }).accepted, false);
assert.equal(submitCall(db, second!.secret, { ok: true, text: "x", model: "claude-sonnet-4-5", usage: { inputTokens: 1, outputTokens: 1 }, durationMs: 1 })?.accepted, false);
assert.equal(await ask, null, "both attempts are used up");
const runs = db.prepare("SELECT state, error FROM agent_runs WHERE prompt_id = 'review.opus' ORDER BY id DESC LIMIT 2").all() as { state: string; error: string }[];
assert.deepEqual(runs.map((x) => x.state), ["failed", "failed"]);
assert.match(runs[1]!.error, /lease expired/);
});
test("three lost leases within two hours cool a token off for six hours; its next claim says why and until when; an operator clears it", async () => {
test("three lost leases of one login within two hours cool its helpers off for six hours; its next claim says why and until when; an operator clears it", async () => {
db.prepare("INSERT INTO access (login, role) VALUES ('erin', 'maintainer')").run();
const erin: Caller = { tokenId: createToken(db, "erin", "helper", false, true).row.id, login: "erin", role: "maintainer" };
const mine = async () => (await claimCall(db, erin, claim, canWrite)).lease;
// A token's drops count once in 10 minutes: three helpers of the same user each lose one.
const tokens = [1, 2, 3].map(() => createToken(db, "erin", "helper", false, true).row.id);
let erin: Caller = { tokenId: tokens[0]!, login: "erin", role: "maintainer" };
const mine = async () => (await claimCall(db, erin, claim(), canWrite)).lease;
const lose = (id: number, now: number) => {
db.prepare("UPDATE remote_calls SET heartbeat_at = ? WHERE id = ?").run(now - 3 * 60000, id);
expireLeases(db, now);
@@ -131,20 +138,23 @@ describe("user-local remote calls", () => {
let lease = null;
for (let i = 0; i < 200 && !lease; i++) { lease = await mine(); if (!lease) await sleep(5); }
lose(lease!.id, t0);
erin = { ...erin, tokenId: tokens[1]! };
const again = (await mine())!;
assert.equal(again.id, lease!.id, "a lost lease is not a failed call: the call goes back to the queue");
lose(again.id, t0 + 1000);
erin = { ...erin, tokenId: tokens[2]! };
assert.equal(await ask, null, "the call used its attempts");
const ask2 = askClaude("review.opus", "opus", SAMPLE_OPUS_PRS, db);
let third = null;
for (let i = 0; i < 200 && !third; i++) { third = await mine(); if (!third) await sleep(5); }
lose(third!.id, t0 + 2000);
const reply = await claimCall(db, erin, claim, canWrite, t0 + 3000);
const reply = await claimCall(db, erin, claim(), canWrite, t0 + 3000);
assert.equal(reply.lease, null);
assert.equal(reply.lease === null && reply.until, t0 + 2000 + 6 * 3600_000);
assert.match(reply.lease === null ? reply.reason ?? "" : "", /3 times within 2 hours/);
assert.equal(clearCooloff(db, "erin", "test"), 1);
assert.ok((await claimCall(db, erin, claim, canWrite, t0 + 3000)).lease, "the call is back in the queue for the cleared helper");
assert.equal((await claimCall(db, { ...erin, tokenId: tokens[0]! }, claim(), canWrite, t0 + 3000)).lease, null, "every token of the login");
assert.equal(clearCooloff(db, "erin", "test"), true);
assert.ok((await claimCall(db, erin, claim(), canWrite, t0 + 3000)).lease, "the call is back in the queue for the cleared helper");
withdrawCall(db, (db.prepare("SELECT id FROM remote_calls WHERE state IN ('queued','leased') ORDER BY id DESC LIMIT 1").get() as { id: number }).id, "test over");
await ask2;
});
@@ -152,15 +162,15 @@ describe("user-local remote calls", () => {
test("a claim whose declared effort is below a tier's minimum gets nothing, and the refusal names the families and the floor", async () => {
setAcceptedModels(db, undefined, "test", { top: "high" });
const claimant = callers.bob!;
await assert.rejects(claimCall(db, claimant, claim, canWrite), /opus\/astra at effort high or higher/);
await assert.rejects(claimCall(db, claimant, { ...claim, efforts: { top: "medium" } }, canWrite), /at effort high or higher/);
assert.equal((await claimCall(db, claimant, { ...claim, efforts: { top: "xhigh" } }, canWrite)).lease, null, "meets the floor; nothing is queued");
await assert.rejects(claimCall(db, claimant, claim(), canWrite), /opus\/astra at effort high or higher/);
await assert.rejects(claimCall(db, claimant, { ...claim(), efforts: { top: "medium" } }, canWrite), /at effort high or higher/);
assert.equal((await claimCall(db, claimant, { ...claim(), efforts: { top: "xhigh" } }, canWrite)).lease, null, "meets the floor; nothing is queued");
setAcceptedModels(db, undefined, "test", { top: "max" });
const codex: Claim = { ...claim, harness: "codex", tiers: { top: "gpt-6-astra" }, efforts: { top: "xhigh" } };
const codex: Claim = { ...claim(), harness: "codex", tiers: { top: "gpt-6-astra" }, efforts: { top: "xhigh" } };
assert.equal((await claimCall(db, claimant, codex, canWrite)).lease, null, "Codex's top level, xhigh, meets a max floor");
await assert.rejects(claimCall(db, claimant, { ...claim, efforts: { top: "xhigh" } }, canWrite), /effort max/);
await assert.rejects(claimCall(db, claimant, { ...claim(), efforts: { top: "xhigh" } }, canWrite), /effort max/);
setAcceptedModels(db, undefined, "test", { top: null });
assert.equal((await claimCall(db, claimant, claim, canWrite)).lease, null, "no floor: any helper");
assert.equal((await claimCall(db, claimant, claim(), canWrite)).lease, null, "no floor: any helper");
});
test("the session tools count what the login may claim, and show a lease only to its holder", async () => {
@@ -203,7 +213,7 @@ describe("user-local remote calls", () => {
await assert.rejects(userSettings.setSettings(db, req("bob", "maintainer", { reserve: { fiveHour: 95, weekly: 10 } })), /0 to 90/);
await assert.rejects(userSettings.setSettings(db, req("eve", "viewer", { reserve: { fiveHour: 30, weekly: 30 } })), /viewers/);
assert.ok(db.prepare("SELECT 1 FROM audit WHERE action = 'usage-reserve' AND login = 'bob'").get());
assert.deepEqual(await claimCall(db, callers.bob!, claim, canWrite), { lease: null, retryAfter: CLAIM_RETRY_S, reserve: { fiveHour: 35, weekly: 10 } });
assert.deepEqual(await claimCall(db, callers.bob!, claim(), canWrite), { lease: null, retryAfter: CLAIM_RETRY_S, reserve: { fiveHour: 35, weekly: 10 } });
assert.equal(remoteRoom(db, 0), 0, "an idle helper that never reported ready");
const status = { slots: 1, busy: 0, harness: "claude", models: { top: "opus" }, reserve: { fiveHour: 35, weekly: 10 }, usage: { fiveHour: { usedPercent: 70, resetsAt: null }, weekly: null, at: new Date().toISOString() } };
storeStatus(db, callers.bob!.tokenId, "bob", parseStatus({ ...status, readiness: "ready" }));
@@ -214,13 +224,13 @@ describe("user-local remote calls", () => {
assert.equal(h?.readiness, "quota-held");
assert.equal(h?.fresh, true);
assert.equal(h?.usage?.fiveHour?.usedPercent, 70);
db.prepare("UPDATE mcp_tokens SET cooloff_until = ? WHERE id = ?").run(Date.now() + 3600_000, callers.bob!.tokenId);
db.prepare("INSERT OR REPLACE INTO remote_cooloffs (login, until, reason) VALUES ('bob', ?, 'test')").run(Date.now() + 3600_000);
assert.equal(listHelpers(db, "bob")[0]?.readiness, "cooled-off", "the server's cool-off shows before the helper reports it");
db.prepare("UPDATE mcp_tokens SET cooloff_until = NULL WHERE id = ?").run(callers.bob!.tokenId);
db.prepare("DELETE FROM remote_cooloffs WHERE login = 'bob'").run();
});
test("a user's repo picks limit their claims and next_task; the helper's own list narrows them, never widens them", async () => {
db.prepare("UPDATE mcp_tokens SET idle_claim_at = NULL, cooloff_until = NULL").run();
db.prepare("UPDATE mcp_tokens SET idle_claim_at = NULL").run();
db.prepare("INSERT INTO tracked_repos (name, pipeline) VALUES ('photocraft', 1), ('fontcraft', 1), ('gridcraft', 0) ON CONFLICT (name) DO UPDATE SET pipeline = excluded.pipeline").run();
const pick = (repos: string[]) => userSettings.setSettings(db, req("bob", "maintainer", { repos }));
const got = await pick(["fontcraft", "gridcraft"]);
@@ -228,17 +238,57 @@ describe("user-local remote calls", () => {
await assert.rejects(pick(["nosuchrepo"]), /not tracked/);
const ask = askClaude("review.opus", "opus", SAMPLE_OPUS_PRS, db);
while (!db.prepare("SELECT 1 FROM remote_calls WHERE state = 'queued'").get()) await sleep(5);
assert.equal((await claimCall(db, callers.bob!, claim, canWrite)).lease, null, "photocraft is not picked");
assert.equal((await claimCall(db, callers.bob!, claim(), canWrite)).lease, null, "photocraft is not picked");
assert.equal((db.prepare("SELECT idle_repos FROM mcp_tokens WHERE id = ?").get(callers.bob!.tokenId) as { idle_repos: string }).idle_repos, '["fontcraft","gridcraft"]', "the idle helper records what it takes");
assert.equal(await claimableCalls(db, callers.bob!, {}, canWrite), 0);
await pick(["photocraft"]);
assert.equal(await claimableCalls(db, callers.bob!, {}, canWrite), 1);
assert.equal((await claimCall(db, callers.bob!, { ...claim, repos: ["fontcraft"] }, canWrite)).lease, null, "the helper's list narrows");
assert.equal((await claimCall(db, callers.bob!, { ...claim(), repos: ["fontcraft"] }, canWrite)).lease, null, "the helper's list narrows");
assert.equal((db.prepare("SELECT idle_repos FROM mcp_tokens WHERE id = ?").get(callers.bob!.tokenId) as { idle_repos: string }).idle_repos, "[]");
const lease = (await claimCall(db, callers.bob!, { ...claim, repos: ["photocraft", "fontcraft"] }, canWrite)).lease!;
const lease = (await claimCall(db, callers.bob!, { ...claim(), repos: ["photocraft", "fontcraft"] }, canWrite)).lease!;
assert.match(lease.title, /photocraft#1287/);
done(lease.secret, "picked");
assert.equal((await ask)?.text, "picked");
await pick([]);
});
const stateOf = (id: number) => (db.prepare("SELECT state FROM remote_calls WHERE id = ?").get(id) as { state: string }).state;
test("a repeated claim gets the same lease, and a dashboard outage ends no lease", async () => {
db.prepare("UPDATE mcp_tokens SET idle_claim_at = NULL").run();
const ask = askClaude("review.opus", "opus", SAMPLE_OPUS_PRS, db);
while (!db.prepare("SELECT 1 FROM remote_calls WHERE state = 'queued'").get()) await sleep(5);
const c = claim();
const [a, b] = await Promise.all([claimCall(db, callers.bob!, c, canWrite), claimCall(db, callers.bob!, c, canWrite)]);
assert.ok(a.lease);
assert.deepEqual(b, a, "a repeat while the first is still being answered");
assert.deepEqual(await claimCall(db, callers.bob!, c, canWrite), a, "a repeat after the reply was lost");
const id = a.lease!.id, now = Date.now();
// The dashboard's last stamp is 5 minutes old; the lease's last heartbeat 3 minutes.
db.prepare("INSERT OR REPLACE INTO remote_liveness (id, alive_at, since) VALUES (1, ?, ?)").run(now - 5 * 60_000, now - 3600_000);
db.prepare("UPDATE remote_calls SET heartbeat_at = ? WHERE id = ?").run(now - 3 * 60_000, id);
expireLeases(db, now);
assert.equal(stateOf(id), "leased", "nothing expires while the dashboard is down");
stampAlive(db, now);
assert.equal((db.prepare("SELECT heartbeat_at FROM remote_calls WHERE id = ?").get(id) as { heartbeat_at: number }).heartbeat_at, now, "the dashboard came back: the heartbeat moved forward by the gap, up to now");
db.prepare("DELETE FROM remote_liveness").run();
done(a.lease!.secret, "after the outage");
assert.equal((await ask)?.text, "after the outage");
});
test("a failed reply sends the call back to the queue with its next attempt, for another user's helper; the old secret answers 410", async () => {
const ask = askClaude("review.opus", "opus", SAMPLE_OPUS_PRS, db);
const first = (await leaseFor("bob"))!;
assert.equal(heartbeat(db, first.secret, { phase: "running" }), true);
assert.deepEqual(submitCall(db, first.secret, { ok: false, error: "harness crashed", durationMs: 5 }), { accepted: true });
assert.deepEqual({ ...db.prepare("SELECT state, attempt, released_by, started_at FROM remote_calls WHERE id = ?").get(first.id) }, { state: "queued", attempt: 2, released_by: "bob", started_at: null });
assert.deepEqual(leaseState(db, first.secret), { id: first.id, live: false });
assert.equal((await claimCall(db, callers.bob!, claim(), canWrite)).lease, null, "not offered to the user whose helper failed it");
assert.equal((db.prepare("SELECT failures FROM remote_cooloffs WHERE login = 'bob'").get() as { failures: number }).failures, 1, "counts toward bob's cool-off");
db.prepare("UPDATE mcp_tokens SET idle_claim_at = NULL").run();
const second = (await leaseFor("alice"))!;
assert.equal(second.id, first.id);
done(second.secret, "second try");
assert.equal((await ask)?.text, "second try");
});
});
+1 -1
View File
@@ -65,7 +65,7 @@ describe("per-lease tools of remote calls", () => {
ended: enqueueCall(db, { ...common, bundle: bundle(["items"]) }),
};
for (let i = 0; i < 4; i++) {
const r = await claimCall(db, caller, { requestId: "test-claim-0001", protocol: PROTOCOL_VERSION, helperVersion: "1.0.0", harness: "claude", harnessVersion: "2", tiers: { top: "opus" } }, async () => true);
const r = await claimCall(db, caller, { requestId: `test-claim-000${i}`, protocol: PROTOCOL_VERSION, helperVersion: "1.1.0", harness: "claude", harnessVersion: "2", tiers: { top: "opus" } }, async () => true);
const name = Object.entries(ids).find(([, id]) => id === r.lease?.id)![0];
leases.set(name, r.lease!);
}