Files
Dylan M. Taylor d4033df996 fix: enforce the bucket request timeout, cap debug sleeps, fix db-cleanup batch paging (#324)
What was wrong:
- LegacyBucketClient called Bucket::set_request_timeout, which rust-s3
(tokio backend) ignores: the HTTP client is built in the constructor. A
storage peer that stalled a response held a worker forever.
- The video-style-transfer, face-fusion and
video-style-transfer-workflow jobs slept for a request-header-supplied
number of milliseconds with no maximum, parking a worker.
- db-cleanup delete_user_files advanced an offset over a listing that
shrinks as files are deleted, so it skipped every other batch.

Change: when a timeout is configured, the bucket is rebuilt with
with_request_timeout (it applies to the whole request, including body).
Debug sleeps go through debug_sleep_duration, capped at 60 s. db-cleanup
always reads offset 0 (deleted files are excluded from the listing).

Blast radius: cloud_storage, inference-job (3 call sites), db-cleanup.

Behavior changes: callers that pass Some(timeout) (jobs default to 5 min
via BUCKET_TIMEOUT_SECONDS, web 60 s-ish via env) now actually get that
limit per request, so very large transfers longer than the configured
value will fail instead of running unbounded. Debug sleeps above 60 s
are clamped.

Checks run: cargo check -p cloud_storage -p db-cleanup -p inference-job;
cargo test -p inference-job --bin inference-job debug_sleep.
Checks not run: fmt/clippy, db-cleanup against a database,
incomplete-download file cleanup on timeout (not changed).


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* File cleanup now processes remaining files in batches without relying
on page offsets.
* Debug delays in face-fusion and video-style-transfer jobs are capped
at 60 seconds, preventing excessively long waits.
* Cloud storage bucket requests now apply a timeout only when one is
configured, with client setup errors propagated.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 11:22:32 -05:00
..