Say in the feed when a request waits for its rate limit (#2158)

An API backend with requests_per_minute set held each request back without a word, so a paced
build looked stuck. A wait of two seconds or more now puts a line in the feed saying how long
it waits and why (#668).
This commit is contained in:
Justin John authored and GitHub committed 2026-11-19 09:26:57 +07:00
1 parent 893aa76646
commit a9130cabed
5 files changed
+56 -9

No files matched your search

+13 -4
View File
@@ -706,8 +706,13 @@ impl AgentLoop {
self.emit(AgentEvent::Status { text: "Making room: summarising the earlier work…".into() });
let messages = context::with_summary_request(&self.history[..cut]);
let req = TurnRequest { system, messages: &messages, tools: &[], effort: Effort::Low, max_tokens: crate::anthropic::MAX_TOKENS };
let stop = self.stop.clone();
let summary = self.chain.get_mut(self.current).map(|slot| slot.model.turn(&req, &mut |_| {}, &stop));
let (stop, events) = (self.stop.clone(), self.events.clone());
let say = &mut |d| {
if let Delta::Status(text) = d {
let _ = events.send(AgentEvent::Status { text });
}
};
let summary = self.chain.get_mut(self.current).map(|slot| slot.model.turn(&req, say, &stop));
let mut accounted = None;
let text = match summary {
Some(Ok(r)) => {
@@ -782,11 +787,15 @@ impl AgentLoop {
let slot = self.chain.get_mut(self.current)?;
let r = slot.model.turn(
&req,
&mut |d| {
if let Delta::Text(t) = d {
&mut |d| match d {
Delta::Text(t) => {
produced = true;
let _ = events.send(AgentEvent::TextDelta { text: t });
}
Delta::Status(text) => {
let _ = events.send(AgentEvent::Status { text });
}
Delta::Thinking(_) => {}
},
&stop,
);
+2
View File
@@ -232,6 +232,8 @@ pub struct TurnResult {
pub enum Delta {
Text(String),
Thinking(String),
/// A line for the feed about the request itself, not the model's words (a wait for its rate limit).
Status(String),
}
/// Cooperative cancellation for a conversation.
+37 -2
View File
@@ -16,6 +16,14 @@ pub(crate) fn gap(requests_per_minute: Option<u32>) -> Option<Duration> {
requests_per_minute.filter(|n| *n > 0).map(|n| Duration::from_secs(60) / n)
}
/// Waits shorter than this pass without a word in the feed.
const SAY_WAIT: Duration = Duration::from_secs(2);
/// What the feed says while a request waits for its backend's rate limit.
fn waiting(backend: &str, wait: Duration) -> String {
format!("Waiting {} s before the next request to {backend}, to stay within its requests-per-minute limit…", wait.as_secs_f64().ceil() as u64)
}
/// A model whose requests start at least `gap` apart.
pub(crate) struct Paced {
pub inner: Box<dyn ChatModel>,
@@ -33,6 +41,11 @@ impl ChatModel for Paced {
next.insert(self.backend.clone(), at + self.gap);
at
};
// A wait the person would notice is said in the feed, so a paced build doesn't look stuck (#668).
let wait = at.saturating_duration_since(Instant::now());
if wait >= SAY_WAIT {
on_delta(Delta::Status(waiting(&self.backend, wait)));
}
// A stop ends the wait; the model itself then answers it as it does any stopped turn.
while !stop.is_stopped()
&& let Some(left) = at.checked_duration_since(Instant::now()).filter(|d| !d.is_zero())
@@ -72,11 +85,33 @@ mod tests {
}
}
fn ask(backend: &str, gap: Duration, stop: &Stop) {
/// Send one request; what it said in the feed while it waited.
fn ask(backend: &str, gap: Duration, stop: &Stop) -> Vec<String> {
let mut paced = Paced { inner: Box::new(Refuses), backend: backend.into(), gap };
assert_eq!(paced.model(), "m");
let req = TurnRequest { system: "", messages: &[], tools: &[], effort: Effort::Low, max_tokens: 1 };
assert_eq!(paced.turn(&req, &mut |_| {}, stop).unwrap_err().message, "asked", "the request is still sent");
let mut said = Vec::new();
let mut on_delta = |d| {
if let Delta::Status(text) = d {
said.push(text);
}
};
assert_eq!(paced.turn(&req, &mut on_delta, stop).unwrap_err().message, "asked", "the request is still sent");
said
}
/// #668: a request held back for its rate limit says so in the feed; one sent at once says nothing.
#[test]
fn a_wait_for_the_limit_is_said() {
assert!(ask("pace-test-said", Duration::from_secs(60), &Stop::default()).is_empty(), "the first goes at once");
let stop = Stop::default();
stop.stop();
assert_eq!(
ask("pace-test-said", Duration::from_secs(60), &stop),
["Waiting 60 s before the next request to pace-test-said, to stay within its requests-per-minute limit…"]
);
assert!(ask("pace-test-short", Duration::from_millis(500), &Stop::default()).is_empty());
assert!(ask("pace-test-short", Duration::from_millis(500), &Stop::default()).is_empty(), "half a second isn't worth a line");
}
#[test]
+3 -2
View File
@@ -550,8 +550,9 @@ been run live.
Not run against a live provider — claimed: justinjohn0306, branch `feature/r5.4-requests-per-minute`
- [x] Offer `requests_per_minute` in Settings → AI and `ai request-rate` (#668)
— PR #1841; claimed: rudycelekli, branch feature/r5.4-request-rate-setting
- [ ] Say in the feed when a build is waiting for its turn (#668); a tokens-per-minute limit
isn't covered either
- [x] Say in the feed when a build is waiting for its turn (#668)
— claimed: justinjohn0306, branch feature/r5.4-say-when-a-request-waits
- [ ] A tokens-per-minute limit isn't covered by the request pacing (#668)
- [x] Find LM Studio on another port (#73): the port in its settings file (when it is there) and
the ports 1235-1239 are probed like the usual servers; OpenCode's own `baseURL`s were already
read (#83)
+1 -1
View File
@@ -452,7 +452,7 @@ coverage and secrets passed. The companion CLI was built with locked dependencie
| Builders that can't see pictures | A builder in the API loop (`BriefCx::sees_pictures` is false; agent apps open pictures with Read) is never sent to open or look at a picture. In its place: `capture.json`'s `layouts` for a website's screenshots, each round's `checks/round-N/capture.json` beside the original's, a program's `capture/ui-tree.json` and the check's likeness, and asking the person with core_person_ask what their pictures show. Agent apps' prompts are unchanged apart from the `--screenshot` contract's last sentence (R5.4) | `a_builder_that_cant_see_is_never_sent_to_a_picture` covers every kind, target, mode and picture case (144) and was shown to fail when a sighted phrase came back; all 150 sighted briefs and refinements were compared byte for byte before and after. No live API-loop build was run after this change |
| API answer time limit | `timeout_secs` in `[[ai.backends]]` also bounds an API backend's streamed answer, waiting for it to start included (five minutes to start and one hour in all when unset or 0), so a slow local model can take longer for a long answer (#1096). A stream that runs out of time says to set it | Unit tests for the limit taken from the setting and for the remediation on a body timeout; no live slow model was run, and a request queued on a shared server (#1239) was not run live |
| AI connection errors | An API backend that took the request and didn't start answering says so, instead of "Couldn't reach". For an address on this computer (`localhost`, `127.x.x.x`, `::1`), the advice says a large model may still be loading or reading a long prompt and that a smaller context length helps, and a failed connection says to check that the server is running on this computer, without the internet advice remote services get | Unit tests for both kinds of failure on local and remote addresses, and for what counts as this computer. Seen live with a local Ollama whose 131,072-token model took over five minutes to start answering (before the change); the new messages were not run live |
| Backend request rate | `requests_per_minute` in `[[ai.backends]]` spaces an API backend's requests out for a service that limits them (120 for two a second): each waits its turn, across every conversation on that backend in the app, and Stop ends the wait. Zero or omission sends them as they come. Set it in Settings → AI, using `ai request-rate <name> [requests-per-minute]`, or in `config.toml`. Coding-agent apps send their own requests and aren't paced | `pace` tests cover the spacing, a second backend not waiting, and a stop; no live provider was used |
| Backend request rate | `requests_per_minute` in `[[ai.backends]]` spaces an API backend's requests out for a service that limits them (120 for two a second): each waits its turn, across every conversation on that backend in the app, and Stop ends the wait. A wait of 2 s or more is said in the feed ("Waiting 30 s before the next request to …"), so a paced build doesn't look stuck (#668). Zero or omission sends them as they come. Set it in Settings → AI, using `ai request-rate <name> [requests-per-minute]`, or in `config.toml`. Coding-agent apps send their own requests and aren't paced | `pace` tests cover the spacing, a second backend not waiting, a stop, and the feed line for a long wait but not a short one; no live provider was used |
| Backend context budgets | `context_tokens` in `[[ai.backends]]` controls when an API conversation is trimmed and summarised. Zero or omission retains local/cloud defaults; a nonzero conversation override takes priority. Before each attempt, including a fallback, fitting adds estimated history growth since the last successful request to its reported input size. Trimming resets that reference, and successful requests refresh it; summary usage remains in message totals. When a server refuses a request for its context window and says the size (Ollama and llama.cpp's "exceeds the available context size", OpenAI-style "maximum context length is"), the error carries it (`context_window`, `prompt_tokens`), the conversation is fitted to three quarters of it (it replaces the local/cloud guess; a set budget stays the most) measured with the server's own count, and the request is sent once more, by each backend in the chain and only when room was made (without the server's count, the refused request is taken as one token over the window); if that fails too, the remediation says how to raise the window (#962). Before a backend's first request, and once more after it while the server hasn't said, an OpenAI-compatible server or OpenRouter is asked for the window it has loaded (`crate::window`: `context_length`, `context_window`, `max_model_len` or `meta.n_ctx` in its model list; one on this computer or its network is also asked through llama-swap's `/upstream/<model>/props` for a loaded model, llama.cpp's `/props`, LM Studio's `loaded_context_length`, Ollama's `/api/ps`); three quarters of it is the budget for a local model, and it only lowers a paid service's 150k default; a budget set on any backend is kept up to the window | Regressions cover settings and model construction, defaults/overrides, smaller fallback budgets, large-budget arithmetic and new tool output growing beyond a fallback budget even when the history estimate alone is below the reported input. The latest tool results are preserved, and a short follow-up does not count already-sent history twice. `a_request_too_long_for_the_servers_window_is_fitted_and_asked_once_more` covers a refit that then fits and one that doesn't (asked once more, never again); `a_refit_makes_room_first_and_each_backend_gets_its_own` covers a refusal without a count, one with nothing to trim, and a fallback refused for its own window; the window is read from the whole message, and OpenAI's prompt part "in the messages", not its total; the server's two wordings are parsed in `http`. This does not set server context parameters, and LM Studio's refusal names no size, so it isn't refitted; no live provider was used for the refit. Each window probe has unit tests on recorded response shapes (an unloaded llama-swap model is never asked through its upstream route, a trained window is not taken for the loaded one, hostile shapes read as nothing); asked live, a llama-swap server read 524288 from its model list for a loaded and an unloaded model and loaded nothing, and a real `chat` through it journaled that window before its first request; its `/upstream/` route read 262144 for a loaded model and for an alias of it, and nothing (loading nothing) for an unloaded one. With Ollama 0.40.15 and `qwen3:0.6b`, a chat before the model was loaded logged no window and one after read 4096 from `/api/ps` (Ollama's default). LM Studio and plain llama.cpp were not run live |
| Builder tool lists | Each API backend's `tools` in `[[ai.backends]]` picks the tools a builder on it is offered. Lean is the building list (#417, `LOCAL_BUILDER_CORE`) and a system prompt without the memory steps; Full is every tool and the memory steps. Auto (the default) is Full on a paid service, and on a local model Full when the conversation's budget is `FULL_TOOLS_BUDGET` (49,152, a 64k window less its quarter) or more, else Lean, so a local model with no window known stays lean as before. The loop asks the server for its window before choosing and chooses again before each turn, rewriting the system prompt when the list changes (journaled as `tools`). Chats keep every tool (R5.4). Choose it in Settings → AI, which shows what Auto works out to for that AI (by its window on a local model, every tool on a paid service), or with `ai tools <name> [auto\|lean\|full]`; Auto is saved as no setting, and agent apps are refused (they choose their own tools) | `ai_tools_reads_changes_and_clears_an_api_backends_tool_list` and `an_api_backends_tool_list_is_chosen_in_settings` cover the CLI and the Settings choice. `a_builders_tool_list_follows_its_setting_and_the_servers_window` covers each setting on local and paid backends, a window named after the first turn (lean, then full), a set budget and a chat; it was seen to fail when the prompt wasn't rewritten and when the server wasn't asked before choosing. Live on Linux, a Qwen model with a 512k window through llama-swap (2× RTX 6000), "a scientific calculator" with no target and one refinement round, unattended: Auto chose the full list (budget 393,216) before the first request. 2 of 3 full-list builds were Ready (22 and 26 minutes, the check's screenshot a working calculator) and used the brief, handoff and finding tools, one also enabling the Runtime toolset to read its own window; the third stopped at check 2 in 3 minutes after the model twice wrote calls to tools that don't exist (`web_search`, `web_fetch`) as text, which ended its turn. The same build on the lean list was Ready in 32 minutes. A small local model wasn't run |
| Games | Engine detection for 12 engines (Unreal, Unity, Godot, Source, GameMaker, RPG Maker, Ren'Py, MonoGame, LÖVE, CryEngine, Cocos, Cube) with versions. Unreal and Godot games are rebuilt in the same engine, others in Unity; installed engines are found, Godot installs itself, others get guided installs. Picking an inner folder (Engine, Binaries, Paks…) finds the game's folder. In a Windows game's folder picked on Linux or macOS, its `.exe` is the program to study (Ghidra in Deep mode), although nothing tries to start it there. Just look inside a game shows what it's made of (`core_target_triage` on its folder: engine and version, how its code ships, the libraries its Unity assembly names give away, its own code, scenes and size) and opens it in the chat to ask about it. The builder keeps `app/MECHANICS.md` and its structured twin `app/mechanics.json` (checked after each build), in Deep mode only, may reuse the original's readable files for private use (never decrypting; generated pictures and 3D models replace its art when the person turns them on; Clean room copies none of its files), lists them in `CREDITS.md` (checked, with the private-use line in `README.md`); a game is ready only when its `--self-test` shows the player moving, acting and the HUD drawn, and the result can be started from the project page | Unit tests; live runs on Windows |