mirror of
https://github.com/storytold/lightcraft.git
synced 2026-11-22 10:21:59 +01:00
* LR-EDIT-DETAIL-DENOISE: AI Denoise works out of the box with a built-in denoiser Turning AI Denoise on no longer needs an ONNX model and a denoise-model.json. Without a model of the user's own, the engine runs a built-in pure-Rust denoiser (noise measured per tile, non-local means on the Bayer planes, edge-directed demosaic) through the same tiling, cache, queue, preview and export path as a model. Installing a model stays optional; Stop using returns to the built-in one, and the id `builtin` is reserved. The set-up prompt and its resume-after-install flow (model_setup.rs, PhotoState::NoModel) are removed. Tests updated for the new behaviour, with new tests for the built-in path; docs, trackers and locale catalogs updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: built-in denoiser cleans the colour planes together and adapts to the noise The built-in denoiser now measures the photo's noise once from the whole mosaic (the model prepares the photo before its tiles run) instead of per tile, stabilises the variance (generalised Anscombe), rotates the four Bayer planes with a Hadamard transform and runs a non-local-means filter whose patch distance spans all planes, so colour blotches are cleaned with the brightness. It cleans gently at low noise to keep fine detail, and harder, with a second pass that matches on the first pass's cleaner picture, as the noise grows. Cached pictures made by the first version are made again (builtin version 2). Adds an opt-in benchmark (tests/denoise_builtin_quality.rs, public CC0 raw plus noise of known level, scored against the original) and tests for the whole-photo noise estimate, strong noise, bad noise values and the strength ramp. On the Sony A7 III sample the result is 3.6, 7.0, 9.1 and 9.7 dB above the noisy picture at light, medium, heavy and extreme noise, against 2.9, 6.0, 6.5 and 5.8 dB for the first version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * contributors: add esli1283 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: docs say how to run the built-in denoiser's quality benchmark Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: built-in denoiser cleans colour further, keeps more texture and leaves no isolated specks Colour-difference planes get a wider pass guided by the cleaned brightness plane, the first-pass threshold is lower (more texture kept), and cells with no matching patch lean on a looser average in heavy noise. Skips negligible weights for speed; bumps builtin::VERSION to 3. Benchmark adds a texture figure; on the public A7 III raw the gain over the noisy picture is +3.7 / +7.2 / +9.7 / +10.4 dB (was +3.6 / +7.0 / +9.1 / +9.7). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Windows: tests build paths one component at a time and doc_path has no unused variable On Windows `cargo xtask ci` failed before it reached any of the work under test: - doc_path.rs bound the gvfs path outside the cfg(not(windows)) block that uses it, an unused variable under -D warnings. - tests_import and the MCP headless discovery test joined "trip/day2" onto a temp folder, giving mixed separators, and compared them with the normalised paths the folder walk returns. They now join each component. Test and lint changes only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: built-in denoiser filters with block-matching collaborative filtering, on the GPU The built-in denoiser's filter is now block matching with collaborative filtering (BM3D / CBM3D, written from the paper) instead of non-local means. On the public Sony A7 III benchmark with added noise it scores 0.25 to 0.6 dB higher at every noise level and keeps more texture at medium to extreme noise (light noise keeps slightly less: 0.58 against 0.62), as fast as before on the processor. docs/denoise.md records the comparison with non-local means and weighted nuclear norm minimisation (tried, not kept: about +0.03 texture for ten times the time) and what the tuning showed. - denoise-core: bm3d.rs (matching on the brightness plane, 2-D DCT plus Haar across the group, hard threshold then Wiener, hard threshold rising with the noise). Builtin::run_with takes the filter stage, so every other step is shared between devices. Tuning.bm3d selects it; non-local means stays as the fallback. builtin::VERSION is 4, so cached pictures are made again. - gpu: bm3d.rs and wgsl/bm3d_*.wgsl run the grouping, the filtering and the putting back on the graphics card, in the same order as the reference (the results agree to rounding: at most 0.1 noise deviation at a few tie blocks, 4e-6 on average, identical every run). Noise measurement, stabilising and the demosaic stay on the processor. FXC (DX12) needs the working arrays at module scope and no dynamically indexed vector: documented in docs/gpu-pipeline.md. - engine: the card is set up once per process for the first photo that may use it, checked against the processor on a check tile (in the filter's own domain: the demosaic's direction choice can flip a pixel on a tiny difference), timed, and used where it is faster. A tile it gets wrong runs on the processor and three stop it. Settings' Run on choice, device line and "Try the graphics card again" now work for the built-in denoiser. Its set-up marker lives in the models folder. - The benchmark takes LC_GPU=1, LC_NLM=1 and LC_B_* for sweeps. On an RTX 4080 SUPER a 256-cell tile takes 22 ms on the card against 408 ms on one processor thread, and the 1536 x 1024 benchmark crop 0.13 to 0.24 s against 1.0 to 2.1 s on four threads, with identical scores. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: the built-in denoiser measures the noise per colour plane and sets its strength from it The first noise estimator read a picture's texture as noise: it never reported less than about 0.003 at mid-grey whatever the real noise, so the denoiser always believed a photo was about as noisy as ISO 12800. The threshold curve was tuned on that over-estimate, and its levels (0.024 to 0.14) are far noisier than real Canon bodies produce (about 0.0025 at ISO 400 up to 0.038 above ISO 25600, measured on a private library; only aggregates are recorded in the docs). - estimate_noise: each colour plane is read on its own (they sit at different levels in a raw), a second-difference operator cancels flat and sloping signal, 12 x 12 blocks, the quietest 5 % at each brightness divided by the bias of that quantile of pure noise (0.67, measured), a constrained line fit. The first version stays as estimate_noise_v1 to compare with. On 30 public pictures with known added noise the 10th to 90th percentile of estimated / true deviation at mid-grey is 0.97 to 1.29 at the ISO 6400 level against 0.94 to 1.96. - bm3d::Params::lambda is a curve of the noise (1.2 at 0.0028 up to 3.5 at 0.14, straight against its logarithm) instead of two end values and a strength, and plan / filter take the noise level; the GPU path shares plan, so CPU and GPU agree as before. Fitted on clean low-ISO photos of a Canon EOS R and R6 Mark III with noise of their measured levels added: the lowest threshold within about a quarter of a decibel of the best at each level. - Result (PSNR, EOS R / R6 Mark III, 10 + 5 photos): ISO 25600 level 38.36 / 38.86 from 36.18 / 36.21, ISO 51200 level 33.83 / 34.45 from 28.42 / 28.78, ISO 6400 39.93 / 40.69 from 39.30 / 39.74, quiet photos no longer over-smoothed; the noise left in the result falls from 1.3 to 3.1 down to 0.73 to 0.99. - Tests: threshold curve, flat versus textured pictures, no-flat-area pictures, clipped / NaN mosaics; an opt-in accuracy harness on the public corpus (denoise_noise_estimate) with a calibration of the quantile bias; the benchmark takes LC_REAL, LC_BIN, LC_FULL, LC_EST and LC_B_LOFF. - builtin::VERSION is 5, so cached pictures are made again. docs/denoise.md has "How the noise is measured" and "Fitted on real cameras". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: GPU workspace pool cannot miss a wake-up; benchmark path and parity date fixed Review findings (CodeRabbit) on the built-in denoiser pull request: - bm3d::Bm3dGpu::forget and nn::NetRunner::forget changed the workspace count without the free-list lock. A take() that had found no free workspace and read made == WORKSPACES, but had not gone to sleep yet, could miss the wake-up and wait for good on a slot that was already free. forget now changes the count under the free lock, in the same order as take(), then notifies. - A test for each runner holds the free lock and checks that forget waits for it; both fail on the old code and pass now. - docs/denoise.md: the benchmark command resolves the raw file from the repository root, since the test runs from the crate's folder. - docs/target-app-parity.md: Last updated matches its newest revision-history entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: doc comments for the functions this branch adds CodeRabbit's pre-merge docstring check counted 49.72 % of the functions this pull request touches as documented (threshold 80 %). Every function the branch adds, the built-in denoiser's block-matching and non-local-means filters, its noise estimator, the GPU runner and their tests, now says what it does or what the test proves. Older functions the branch only touched are left as they were. Comments only: no code line changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: make the GPU set-up marker's folder; fix the benchmark's default path The crash guard was lost when the models folder did not exist yet. The opt-in quality benchmark resolved its default raw file against crates/engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: enforce the cache limit without a models folder The built-in denoiser makes products when there is no models folder, but eviction returned early in that case, so the cache could outgrow its limit. It now applies the default limit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * LR-EDIT-DETAIL-DENOISE: the cache-limit test needs no 20 GiB file off Unix The no-models-folder test now checks the limit itself on every platform and keeps the oversized sparse file for Unix only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * S0002: review fixes for the built-in denoiser Tell the runner a photo's noise only when it could be measured, run prepare() after the work gate with a cancel check, reword the Settings hint for builds without a models folder, drop untranslated placeholder locale entries (they fall back to English) and restore the cs.json spacing; rustfmt. Co-Authored-By: Claude Code <noreply@anthropic.com> * S0002: fix review findings: command descriptions, stale gpu test, self-test noise The denoise.models.select and denoise.toggle descriptions now describe the built-in fallback. Drop our workspace-forget test, which read fields that upstream's Workspaces pool (#2631, same fix) replaced and so did not compile; upstream's own tests cover it. The built-in self-test is told a noise instead of measuring a 16-cell tile too small to measure. Co-Authored-By: Claude Code <noreply@anthropic.com> --------- Co-authored-by: esli1283 <esli.aaron@outlook.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com>
1228 lines
52 KiB
Rust
1228 lines
52 KiB
Rust
//! Denoise networks on the GPU: a [`NetRunner`] executes a [`Net`] (the plain description of a U-Net's layers) with
|
||
//! WGSL compute kernels, one tile at a time, behind the same [`TileRunner`] interface the CPU runner has.
|
||
//!
|
||
//! The runner has a device of its own, so a denoise job and the interactive renders are separate contexts that the
|
||
//! driver time-slices, and a failure of one never takes the other down. It respects the same switches as GPU rendering
|
||
//! (`LIGHTCRAFT_GPU`, `LIGHTCRAFT_GPU_BACKEND`, the GPU preference) and the same crash sentinel around device creation.
|
||
//! `LIGHTCRAFT_GPU_ADAPTER` picks an adapter by (part of) its name, e.g. to try another card (`backend::pick_adapter`,
|
||
//! shared with the render device).
|
||
//!
|
||
//! Every failure is an `Err` (never a panic), and a device that errs or is lost stops being used: callers then run the
|
||
//! CPU runner. The kernels (`wgsl/nn_conv.wgsl`, `wgsl/nn_pool.wgsl`) are tested against the plain-loop reference
|
||
//! interpreter in `lightcraft_denoise::reference` and, with the real model, against the pure-Rust CPU runner.
|
||
|
||
use std::collections::HashMap;
|
||
use std::sync::atomic::{AtomicBool, Ordering};
|
||
use std::sync::{Condvar, Mutex, OnceLock};
|
||
use std::time::Duration;
|
||
|
||
use bytemuck::{Pod, Zeroable};
|
||
use lightcraft_denoise::net::{Net, Op};
|
||
use lightcraft_denoise::run::{Error, TileRunner};
|
||
use wgpu::util::DeviceExt;
|
||
|
||
/// Longest wait for one tile's GPU work before the device is given up on.
|
||
pub(crate) const TIMEOUT: Duration = Duration::from_secs(60);
|
||
/// The largest tile side (in input cells) the dispatch sizes and 32-bit indexing are checked for.
|
||
const MAX_TILE: usize = 1024;
|
||
/// The largest single buffer used: indices in the kernels are 32-bit element offsets.
|
||
const MAX_BUFFER: u64 = 2 << 30;
|
||
/// Tiles that can be on the GPU at once (each has buffers of its own).
|
||
const WORKSPACES: usize = 2;
|
||
|
||
// ---------------------------------------------------------------------------------------------------------------------
|
||
// the device
|
||
|
||
pub(crate) struct Dev {
|
||
pub(crate) device: wgpu::Device,
|
||
pub(crate) queue: wgpu::Queue,
|
||
pub(crate) info: wgpu::AdapterInfo,
|
||
pub(crate) limits: wgpu::Limits,
|
||
}
|
||
|
||
static DEV: OnceLock<Result<Dev, String>> = OnceLock::new();
|
||
static BROKEN: AtomicBool = AtomicBool::new(false);
|
||
static BROKEN_WHY: Mutex<Option<String>> = Mutex::new(None);
|
||
|
||
pub(crate) fn mark_broken(why: &str) {
|
||
let mut w = BROKEN_WHY.lock().unwrap_or_else(|e| e.into_inner());
|
||
if w.is_none() {
|
||
*w = Some(why.to_string());
|
||
}
|
||
BROKEN.store(true, Ordering::Relaxed);
|
||
}
|
||
|
||
/// Why the GPU stopped being used for denoise (a device error), if it did.
|
||
pub fn broken_reason() -> Option<String> {
|
||
BROKEN.load(Ordering::Relaxed).then(|| BROKEN_WHY.lock().unwrap_or_else(|e| e.into_inner()).clone().unwrap_or_else(|| "device error".into()))
|
||
}
|
||
|
||
pub(crate) fn dev() -> Result<&'static Dev, String> {
|
||
if let Some(why) = broken_reason() {
|
||
return Err(format!("stopped after a GPU failure: {why}"));
|
||
}
|
||
if let Some(why) = crate::switched_off() {
|
||
return Err(why);
|
||
}
|
||
// creating the device is GPU work too: not once the process is ending (issue #620)
|
||
let _work = match DEV.get() {
|
||
Some(_) => None,
|
||
None => Some(crate::exit::enter().ok_or(CLOSING)?),
|
||
};
|
||
DEV.get_or_init(create_device).as_ref().map_err(|e| e.clone())
|
||
}
|
||
|
||
/// Why nothing runs on the GPU once [`crate::begin_shutdown`] was called.
|
||
pub(crate) const CLOSING: &str = "LightCraft is closing";
|
||
|
||
fn create_device() -> Result<Dev, String> {
|
||
let Some(backends) = crate::backend::compute_backends() else { return Err("disabled by LIGHTCRAFT_GPU_BACKEND=off".into()) };
|
||
crate::backend::with_init_marker(backends, || {
|
||
std::panic::catch_unwind(|| make_device(backends)).unwrap_or_else(|_| Err("device creation panicked".into()))
|
||
})
|
||
}
|
||
|
||
fn make_device(backends: wgpu::Backends) -> Result<Dev, String> {
|
||
let mut desc = wgpu::InstanceDescriptor::new_without_display_handle();
|
||
desc.backends = backends;
|
||
// DX12 shaders compile with FXC (issue #471)
|
||
desc.backend_options = crate::backend::backend_options();
|
||
// no indirect-call validation pipeline: Intel Metal fails to compile it (#250)
|
||
desc.flags = crate::backend::instance_flags_from_env(desc.flags);
|
||
let instance = wgpu::Instance::new(desc);
|
||
let adapter = crate::backend::pick_adapter(&instance, backends)?;
|
||
let info = adapter.get_info();
|
||
if info.device_type == wgpu::DeviceType::Cpu {
|
||
return Err(format!("software adapter ({}) skipped: the CPU is faster", info.name));
|
||
}
|
||
let limits = adapter.limits();
|
||
if limits.max_storage_buffers_per_shader_stage < 6
|
||
|| limits.max_compute_invocations_per_workgroup < 256
|
||
|| limits.max_compute_workgroup_size_x < 256
|
||
{
|
||
return Err(format!("{} is too limited for the denoise kernels", info.name));
|
||
}
|
||
if limits.max_compute_workgroup_storage_size < 16 * 1024 {
|
||
return Err(format!("{} has too little workgroup memory for the denoise kernels", info.name));
|
||
}
|
||
let (device, queue) = pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor {
|
||
label: Some("lightcraft-denoise"),
|
||
required_limits: limits.clone(),
|
||
..Default::default()
|
||
}))
|
||
.map_err(|e| format!("{}: device creation failed ({e})", info.name))?;
|
||
device.on_uncaptured_error(std::sync::Arc::new(|e| {
|
||
log::error!("gpu (denoise): {e}");
|
||
mark_broken(&format!("device error: {e}"));
|
||
}));
|
||
device.set_device_lost_callback(|reason, msg| {
|
||
log::error!("gpu (denoise): device lost ({reason:?}): {msg}");
|
||
mark_broken(&format!("device lost ({reason:?}): {msg}"));
|
||
});
|
||
log::info!("gpu (denoise): {} ({:?})", info.name, info.backend);
|
||
Ok(Dev { device, queue, info, limits })
|
||
}
|
||
|
||
/// The adapter denoise would use ("NVIDIA GeForce RTX 4090 Laptop GPU (Dx12)"), or why it cannot use one. Creates the
|
||
/// device on first use.
|
||
pub fn adapter() -> Result<String, String> {
|
||
dev().map(|d| format!("{} ({:?})", d.info.name, d.info.backend))
|
||
}
|
||
|
||
// ---------------------------------------------------------------------------------------------------------------------
|
||
// the plan: what to run, in what buffers
|
||
|
||
#[repr(C)]
|
||
#[derive(Clone, Copy, Pod, Zeroable)]
|
||
struct ConvParams {
|
||
h: u32,
|
||
w: u32,
|
||
c0: u32,
|
||
c1: u32,
|
||
n: u32,
|
||
npad: u32,
|
||
k: u32,
|
||
mode: u32,
|
||
cout: u32,
|
||
leaky: u32,
|
||
alpha: f32,
|
||
/// Always 0: the kernel steps its output stores by `1 + zero`, so FXC cannot merge them (`nn_conv.wgsl`).
|
||
zero: u32,
|
||
}
|
||
|
||
#[repr(C)]
|
||
#[derive(Clone, Copy, Pod, Zeroable)]
|
||
struct PoolParams {
|
||
h: u32,
|
||
w: u32,
|
||
c4: u32,
|
||
total: u32,
|
||
}
|
||
|
||
#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
|
||
enum Kernel {
|
||
Conv { bm: u32, bn: u32, bk: u32 },
|
||
Pool,
|
||
}
|
||
|
||
/// Where a step's tensor lives.
|
||
#[derive(Clone, Copy, Debug)]
|
||
enum Place {
|
||
Input,
|
||
Slot(usize),
|
||
Output,
|
||
}
|
||
|
||
struct Step {
|
||
kernel: Kernel,
|
||
groups: [u32; 3],
|
||
src: Place,
|
||
src2: Option<Place>,
|
||
dst: Place,
|
||
/// Index into the per-layer GPU data (weights, bias), for the convolutions.
|
||
layer: Option<usize>,
|
||
params: Vec<u8>,
|
||
}
|
||
|
||
struct Layout {
|
||
steps: Vec<Step>,
|
||
/// Bytes of each activation buffer.
|
||
slots: Vec<u64>,
|
||
in_floats: usize,
|
||
out_floats: usize,
|
||
}
|
||
|
||
/// Weights arranged for the kernel: `[k][npad]` rows with `k = tap · cin + ci` (columns are output channels, zero past
|
||
/// `n`), and the bias padded the same.
|
||
struct Layer {
|
||
weight: Vec<f32>,
|
||
bias: Vec<f32>,
|
||
}
|
||
|
||
fn unsupported<T>(why: impl Into<String>) -> Result<T, String> {
|
||
Err(why.into())
|
||
}
|
||
|
||
/// The kernel shape for `n` output columns: 64 × 64 blocks when `n` is a multiple of 64, else 128 × 32.
|
||
fn blocks(n: usize, cin: usize) -> (u32, u32, u32) {
|
||
let bn = if n.is_multiple_of(64) { 64 } else { 32 };
|
||
let bk = if cin.is_multiple_of(16) {
|
||
16
|
||
} else if cin.is_multiple_of(8) {
|
||
8
|
||
} else {
|
||
4
|
||
};
|
||
(4096 / bn, bn, bk)
|
||
}
|
||
|
||
fn round_up(v: usize, to: usize) -> usize {
|
||
v.div_ceil(to) * to
|
||
}
|
||
|
||
fn validate_plan(net: &Net, tile: usize, ops: &[Op]) -> Result<(), String> {
|
||
if !net.fits_tile(tile) {
|
||
return unsupported("the tile does not fit the network");
|
||
}
|
||
if tile > MAX_TILE {
|
||
return unsupported("the tile is larger than the kernels' dispatches reach");
|
||
}
|
||
if !net.in_channels.is_multiple_of(4) {
|
||
return unsupported("the input channel count is not a multiple of 4");
|
||
}
|
||
let count = ops.len();
|
||
let ends_right = matches!(
|
||
(count.checked_sub(2).and_then(|i| ops.get(i)), ops.last()),
|
||
(Some(Op::Conv(conv)), Some(Op::DepthToSpace2 { src }))
|
||
if conv.k == 1 && conv.src2.is_none() && *src == count - 1 && conv.cout == 12 && conv.leaky.is_none()
|
||
);
|
||
if ends_right { Ok(()) } else { unsupported("the network does not end in a 1 × 1 convolution to 12 channels and depth-to-space") }
|
||
}
|
||
|
||
/// Work out the steps, the buffers and the arranged weights for `net` on `tile × tile` input cells.
|
||
fn plan(net: &Net, tile: usize, max_binding: u64) -> Result<(Layout, Vec<Layer>), String> {
|
||
let ops = net.ops();
|
||
validate_plan(net, tile, ops)?;
|
||
let side = |t: usize| -> usize {
|
||
let level = net.shape(t).map_or(0, |s| s.level);
|
||
if level >= 0 { tile >> level } else { tile << (-level) }
|
||
};
|
||
let chan = |t: usize| net.shape(t).map_or(0, |s| s.channels);
|
||
let n_ops = ops.len();
|
||
// the last layer that reads each tensor
|
||
let mut last_use = vec![0usize; n_ops + 1];
|
||
for (i, op) in ops.iter().enumerate() {
|
||
let reads: Vec<usize> = match op {
|
||
Op::Conv(c) => std::iter::once(c.src).chain(c.src2).collect(),
|
||
Op::ConvT2(c) => vec![c.src],
|
||
Op::MaxPool2 { src } | Op::DepthToSpace2 { src } => vec![*src],
|
||
};
|
||
for t in reads {
|
||
if let Some(l) = last_use.get_mut(t) {
|
||
*l = i;
|
||
}
|
||
}
|
||
}
|
||
|
||
use Place::{Input, Output, Slot};
|
||
let mut slots: Vec<u64> = Vec::new();
|
||
let mut free: Vec<usize> = Vec::new();
|
||
let mut place: Vec<Option<Place>> = vec![None; n_ops + 1];
|
||
if let Some(p) = place.first_mut() {
|
||
*p = Some(Input);
|
||
}
|
||
let mut steps: Vec<Step> = Vec::new();
|
||
let mut layers: Vec<Layer> = Vec::new();
|
||
/// A buffer of at least `bytes`: the smallest free one that is big enough, else the biggest free one grown, else new.
|
||
fn alloc(bytes: u64, slots: &mut Vec<u64>, free: &mut Vec<usize>) -> usize {
|
||
let size = |s: usize| slots.get(s).copied().unwrap_or(0);
|
||
let fits = free.iter().enumerate().filter(|&(_, &s)| size(s) >= bytes).min_by_key(|&(_, &s)| size(s)).map(|(i, _)| i);
|
||
let pick = fits.or_else(|| free.iter().enumerate().max_by_key(|&(_, &s)| size(s)).map(|(i, _)| i));
|
||
match pick {
|
||
Some(i) => {
|
||
let s = free.swap_remove(i);
|
||
if let Some(b) = slots.get_mut(s) {
|
||
*b = (*b).max(bytes);
|
||
}
|
||
s
|
||
}
|
||
None => {
|
||
slots.push(bytes);
|
||
slots.len() - 1
|
||
}
|
||
}
|
||
}
|
||
let get = |place: &[Option<Place>], t: usize| place.get(t).copied().flatten().ok_or_else(|| format!("tensor {t} has no buffer"));
|
||
|
||
for (i, op) in ops.iter().enumerate() {
|
||
let out_t = i + 1;
|
||
match op {
|
||
Op::Conv(c) => {
|
||
if !c.cin.is_multiple_of(4) || !c.cout.is_multiple_of(4) {
|
||
return unsupported("a convolution whose channel counts are not multiples of 4");
|
||
}
|
||
let s = side(c.src);
|
||
let last = i + 2 == n_ops;
|
||
let (n, mode) = (c.cout, if last { 2u32 } else { 0u32 });
|
||
let (bm, bn, bk) = blocks(n, c.cin);
|
||
let npad = round_up(n, bn as usize);
|
||
// arranged weights: row (tap · cin + ci), column co
|
||
let taps = c.k * c.k;
|
||
let mut weight = vec![0f32; taps * c.cin * npad];
|
||
for co in 0..c.cout {
|
||
for ci in 0..c.cin {
|
||
for ky in 0..c.k {
|
||
for kx in 0..c.k {
|
||
let v = c.weight.get(((co * c.cin + ci) * c.k + ky) * c.k + kx).copied().unwrap_or(0.0);
|
||
if let Some(w) = weight.get_mut(((ky * c.k + kx) * c.cin + ci) * npad + co) {
|
||
*w = v;
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
let mut bias = vec![0f32; npad];
|
||
if let Some(b) = bias.get_mut(..c.cout) {
|
||
b.copy_from_slice(&c.bias);
|
||
}
|
||
let m = s * s;
|
||
let params = ConvParams {
|
||
h: s as u32,
|
||
w: s as u32,
|
||
c0: chan(c.src) as u32,
|
||
c1: c.src2.map_or(0, chan) as u32,
|
||
n: n as u32,
|
||
npad: npad as u32,
|
||
k: c.k as u32,
|
||
mode,
|
||
cout: c.cout as u32,
|
||
leaky: u32::from(c.leaky.is_some()),
|
||
alpha: c.leaky.unwrap_or(0.0),
|
||
zero: 0,
|
||
};
|
||
let src = get(&place, c.src)?;
|
||
let src2 = c.src2.map(|t| get(&place, t)).transpose()?;
|
||
let dst = if last {
|
||
Output
|
||
} else {
|
||
let slot = alloc((m * c.cout * 4) as u64, &mut slots, &mut free);
|
||
Slot(slot)
|
||
};
|
||
if let Some(p) = place.get_mut(out_t) {
|
||
*p = Some(dst);
|
||
}
|
||
layers.push(Layer { weight, bias });
|
||
steps.push(Step {
|
||
kernel: Kernel::Conv { bm, bn, bk },
|
||
groups: [m.div_ceil(bm as usize) as u32, (npad as u32) / bn, 1],
|
||
src,
|
||
src2,
|
||
dst,
|
||
layer: Some(layers.len() - 1),
|
||
params: bytemuck::bytes_of(¶ms).to_vec(),
|
||
});
|
||
}
|
||
Op::ConvT2(c) => {
|
||
if !c.cin.is_multiple_of(4) || !c.cout.is_multiple_of(4) {
|
||
return unsupported("a transposed convolution whose channel counts are not multiples of 4");
|
||
}
|
||
let s = side(c.src);
|
||
let n = 4 * c.cout;
|
||
let (bm, bn, bk) = blocks(n, c.cin);
|
||
let npad = round_up(n, bn as usize);
|
||
// arranged weights: row ci, column (ky · 2 + kx) · cout + co
|
||
let mut weight = vec![0f32; c.cin * npad];
|
||
for ci in 0..c.cin {
|
||
for co in 0..c.cout {
|
||
for ky in 0..2 {
|
||
for kx in 0..2 {
|
||
let v = c.weight.get(((ci * c.cout + co) * 2 + ky) * 2 + kx).copied().unwrap_or(0.0);
|
||
if let Some(w) = weight.get_mut(ci * npad + (ky * 2 + kx) * c.cout + co) {
|
||
*w = v;
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
let mut bias = vec![0f32; npad];
|
||
for q in 0..4 {
|
||
for co in 0..c.cout {
|
||
if let (Some(b), Some(v)) = (bias.get_mut(q * c.cout + co), c.bias.get(co)) {
|
||
*b = *v;
|
||
}
|
||
}
|
||
}
|
||
let m = s * s;
|
||
let params = ConvParams {
|
||
h: s as u32,
|
||
w: s as u32,
|
||
c0: c.cin as u32,
|
||
c1: 0,
|
||
n: n as u32,
|
||
npad: npad as u32,
|
||
k: 1,
|
||
mode: 1,
|
||
cout: c.cout as u32,
|
||
leaky: u32::from(c.leaky.is_some()),
|
||
alpha: c.leaky.unwrap_or(0.0),
|
||
zero: 0,
|
||
};
|
||
let src = get(&place, c.src)?;
|
||
let slot = alloc((4 * m * c.cout * 4) as u64, &mut slots, &mut free);
|
||
if let Some(p) = place.get_mut(out_t) {
|
||
*p = Some(Slot(slot));
|
||
}
|
||
layers.push(Layer { weight, bias });
|
||
steps.push(Step {
|
||
kernel: Kernel::Conv { bm, bn, bk },
|
||
groups: [m.div_ceil(bm as usize) as u32, (npad as u32) / bn, 1],
|
||
src,
|
||
src2: None,
|
||
dst: Slot(slot),
|
||
layer: Some(layers.len() - 1),
|
||
params: bytemuck::bytes_of(¶ms).to_vec(),
|
||
});
|
||
}
|
||
Op::MaxPool2 { src } => {
|
||
let (s, c) = (side(*src), chan(*src));
|
||
if !c.is_multiple_of(4) {
|
||
return unsupported("a max-pool over a channel count that is not a multiple of 4");
|
||
}
|
||
let total = (s / 2) * (s / 2) * (c / 4);
|
||
let params = PoolParams { h: s as u32, w: s as u32, c4: (c / 4) as u32, total: total as u32 };
|
||
let from = get(&place, *src)?;
|
||
let slot = alloc((total * 16) as u64, &mut slots, &mut free);
|
||
if let Some(p) = place.get_mut(out_t) {
|
||
*p = Some(Slot(slot));
|
||
}
|
||
let groups = total.div_ceil(64);
|
||
steps.push(Step {
|
||
kernel: Kernel::Pool,
|
||
groups: [groups.min(8192) as u32, groups.div_ceil(8192) as u32, 1],
|
||
src: from,
|
||
src2: None,
|
||
dst: Slot(slot),
|
||
layer: None,
|
||
params: bytemuck::bytes_of(¶ms).to_vec(),
|
||
});
|
||
}
|
||
// run as part of the 1 × 1 convolution before it
|
||
Op::DepthToSpace2 { .. } => {}
|
||
}
|
||
// buffers whose last reader was this layer are free for the next layer's result
|
||
for t in 0..=i {
|
||
if last_use.get(t).copied() == Some(i)
|
||
&& let Some(Some(Slot(s))) = place.get(t)
|
||
&& !free.contains(s)
|
||
{
|
||
free.push(*s);
|
||
}
|
||
}
|
||
}
|
||
if slots.iter().any(|&b| b > max_binding) {
|
||
return unsupported("an activation buffer is larger than this GPU allows");
|
||
}
|
||
if layers.iter().any(|l| (l.weight.len() as u64).saturating_mul(4) > max_binding) {
|
||
return unsupported("a weight buffer is larger than this GPU allows");
|
||
}
|
||
Ok((Layout { steps, slots, in_floats: net.in_channels * tile * tile, out_floats: 3 * (2 * tile) * (2 * tile) }, layers))
|
||
}
|
||
|
||
// ---------------------------------------------------------------------------------------------------------------------
|
||
// the GPU objects
|
||
|
||
struct Pipelines {
|
||
conv: HashMap<(u32, u32, u32), wgpu::ComputePipeline>,
|
||
conv_layout: wgpu::BindGroupLayout,
|
||
pool: wgpu::ComputePipeline,
|
||
pool_layout: wgpu::BindGroupLayout,
|
||
/// How long each kernel took to build (the driver compiles them), for the log and the tests.
|
||
built: Vec<(String, f64)>,
|
||
}
|
||
|
||
pub(crate) fn entry(binding: u32, ty: wgpu::BufferBindingType) -> wgpu::BindGroupLayoutEntry {
|
||
wgpu::BindGroupLayoutEntry {
|
||
binding,
|
||
visibility: wgpu::ShaderStages::COMPUTE,
|
||
ty: wgpu::BindingType::Buffer { ty, has_dynamic_offset: false, min_binding_size: None },
|
||
count: None,
|
||
}
|
||
}
|
||
|
||
pub(crate) fn pipeline(d: &Dev, label: &str, src: &str, layout: &wgpu::BindGroupLayout) -> wgpu::ComputePipeline {
|
||
let module = d.device.create_shader_module(wgpu::ShaderModuleDescriptor { label: Some(label), source: wgpu::ShaderSource::Wgsl(src.into()) });
|
||
let pl = d.device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor { label: None, bind_group_layouts: &[Some(layout)], immediate_size: 0 });
|
||
d.device.create_compute_pipeline(&wgpu::ComputePipelineDescriptor {
|
||
label: Some(label),
|
||
layout: Some(&pl),
|
||
module: &module,
|
||
entry_point: Some("main"),
|
||
// The kernels write all of their workgroup memory before reading any of it, so the zeroing wgpu adds by default
|
||
// is not needed; on DX12 it is written out as one assignment per element, which makes the shader compiler take
|
||
// a minute over a 2k-element array.
|
||
compilation_options: wgpu::PipelineCompilationOptions { zero_initialize_workgroup_memory: false, ..Default::default() },
|
||
cache: None,
|
||
})
|
||
}
|
||
|
||
fn conv_source(bm: u32, bn: u32, bk: u32) -> String {
|
||
include_str!("wgsl/nn_conv.wgsl")
|
||
.replace("{{BM}}", &bm.to_string())
|
||
.replace("{{BN}}", &bn.to_string())
|
||
.replace("{{BK}}", &bk.to_string())
|
||
.replace("{{B_VECS}}", &(bk * bn / 4).to_string())
|
||
}
|
||
|
||
fn conv_bgl(d: &Dev) -> wgpu::BindGroupLayout {
|
||
use wgpu::BufferBindingType::{Storage, Uniform};
|
||
d.device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
|
||
label: None,
|
||
entries: &[
|
||
entry(0, Uniform),
|
||
entry(1, Storage { read_only: true }),
|
||
entry(2, Storage { read_only: true }),
|
||
entry(3, Storage { read_only: true }),
|
||
entry(4, Storage { read_only: true }),
|
||
entry(5, Storage { read_only: false }),
|
||
],
|
||
})
|
||
}
|
||
|
||
fn pool_bgl(d: &Dev) -> wgpu::BindGroupLayout {
|
||
use wgpu::BufferBindingType::{Storage, Uniform};
|
||
d.device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
|
||
label: None,
|
||
entries: &[entry(0, Uniform), entry(1, Storage { read_only: true }), entry(2, Storage { read_only: false })],
|
||
})
|
||
}
|
||
|
||
fn build_pipelines(d: &Dev, layout: &Layout) -> Pipelines {
|
||
let (conv_layout, pool_layout) = (conv_bgl(d), pool_bgl(d));
|
||
let mut conv = HashMap::new();
|
||
let mut built = Vec::new();
|
||
for s in &layout.steps {
|
||
if let Kernel::Conv { bm, bn, bk } = s.kernel {
|
||
conv.entry((bm, bn, bk)).or_insert_with(|| {
|
||
let started = std::time::Instant::now();
|
||
let p = pipeline(d, "nn_conv", &conv_source(bm, bn, bk), &conv_layout);
|
||
built.push((format!("conv {bm}x{bn}x{bk}"), started.elapsed().as_secs_f64() * 1000.0));
|
||
p
|
||
});
|
||
}
|
||
}
|
||
let started = std::time::Instant::now();
|
||
let pool = pipeline(d, "nn_pool", include_str!("wgsl/nn_pool.wgsl"), &pool_layout);
|
||
built.push(("pool".into(), started.elapsed().as_secs_f64() * 1000.0));
|
||
Pipelines { conv, conv_layout, pool, pool_layout, built }
|
||
}
|
||
|
||
/// One tile's buffers and the bound steps.
|
||
struct Workspace {
|
||
input: wgpu::Buffer,
|
||
output: wgpu::Buffer,
|
||
staging: wgpu::Buffer,
|
||
bound: Vec<(Kernel, [u32; 3], wgpu::BindGroup)>,
|
||
_slots: Vec<wgpu::Buffer>,
|
||
}
|
||
|
||
struct Shared {
|
||
dev: &'static Dev,
|
||
layout: Layout,
|
||
pipes: Pipelines,
|
||
weights: Vec<(wgpu::Buffer, wgpu::Buffer)>,
|
||
uniforms: Vec<wgpu::Buffer>,
|
||
dummy: wgpu::Buffer,
|
||
}
|
||
|
||
pub(crate) fn storage(d: &Dev, label: &str, bytes: u64, extra: wgpu::BufferUsages) -> wgpu::Buffer {
|
||
d.device.create_buffer(&wgpu::BufferDescriptor {
|
||
label: Some(label),
|
||
size: bytes.max(16),
|
||
usage: wgpu::BufferUsages::STORAGE | extra,
|
||
mapped_at_creation: false,
|
||
})
|
||
}
|
||
|
||
impl Shared {
|
||
fn workspace(&self) -> Workspace {
|
||
let d = self.dev;
|
||
let slots: Vec<wgpu::Buffer> = self.layout.slots.iter().map(|&b| storage(d, "activations", b, wgpu::BufferUsages::empty())).collect();
|
||
let input = storage(d, "input", (self.layout.in_floats * 4) as u64, wgpu::BufferUsages::COPY_DST);
|
||
let output = storage(d, "output", (self.layout.out_floats * 4) as u64, wgpu::BufferUsages::COPY_SRC);
|
||
let staging = d.device.create_buffer(&wgpu::BufferDescriptor {
|
||
label: Some("readback"),
|
||
size: (self.layout.out_floats * 4) as u64,
|
||
usage: wgpu::BufferUsages::MAP_READ | wgpu::BufferUsages::COPY_DST,
|
||
mapped_at_creation: false,
|
||
});
|
||
let buf = |p: Place| -> &wgpu::Buffer {
|
||
match p {
|
||
Place::Input => &input,
|
||
Place::Output => &output,
|
||
Place::Slot(i) => slots.get(i).unwrap_or(&self.dummy),
|
||
}
|
||
};
|
||
let mut bound = Vec::new();
|
||
for (i, s) in self.layout.steps.iter().enumerate() {
|
||
let uniform = self.uniforms.get(i).unwrap_or(&self.dummy);
|
||
let bg = match s.kernel {
|
||
Kernel::Conv { .. } => {
|
||
let (w, b) = s.layer.and_then(|l| self.weights.get(l)).map_or((&self.dummy, &self.dummy), |(w, b)| (w, b));
|
||
d.device.create_bind_group(&wgpu::BindGroupDescriptor {
|
||
label: None,
|
||
layout: &self.pipes.conv_layout,
|
||
entries: &[
|
||
wgpu::BindGroupEntry { binding: 0, resource: uniform.as_entire_binding() },
|
||
wgpu::BindGroupEntry { binding: 1, resource: buf(s.src).as_entire_binding() },
|
||
wgpu::BindGroupEntry { binding: 2, resource: s.src2.map_or(&self.dummy, buf).as_entire_binding() },
|
||
wgpu::BindGroupEntry { binding: 3, resource: w.as_entire_binding() },
|
||
wgpu::BindGroupEntry { binding: 4, resource: b.as_entire_binding() },
|
||
wgpu::BindGroupEntry { binding: 5, resource: buf(s.dst).as_entire_binding() },
|
||
],
|
||
})
|
||
}
|
||
Kernel::Pool => d.device.create_bind_group(&wgpu::BindGroupDescriptor {
|
||
label: None,
|
||
layout: &self.pipes.pool_layout,
|
||
entries: &[
|
||
wgpu::BindGroupEntry { binding: 0, resource: uniform.as_entire_binding() },
|
||
wgpu::BindGroupEntry { binding: 1, resource: buf(s.src).as_entire_binding() },
|
||
wgpu::BindGroupEntry { binding: 2, resource: buf(s.dst).as_entire_binding() },
|
||
],
|
||
}),
|
||
};
|
||
bound.push((s.kernel, s.groups, bg));
|
||
}
|
||
Workspace { input, output, staging, bound, _slots: slots }
|
||
}
|
||
}
|
||
|
||
/// Workspaces not in use, and how many exist (free or with a tile in flight).
|
||
struct Slots<T> {
|
||
free: Vec<T>,
|
||
made: usize,
|
||
}
|
||
|
||
/// Up to `cap` workspaces shared by the threads that run tiles. The free list and the count share one lock, so a caller
|
||
/// that finds none free and waits cannot miss one being given back or forgotten in between (issue #2577).
|
||
struct Workspaces<T> {
|
||
slots: Mutex<Slots<T>>,
|
||
wake: Condvar,
|
||
cap: usize,
|
||
}
|
||
|
||
impl<T> Workspaces<T> {
|
||
fn new(first: T, cap: usize) -> Self {
|
||
Workspaces { slots: Mutex::new(Slots { free: vec![first], made: 1 }), wake: Condvar::new(), cap }
|
||
}
|
||
|
||
fn lock(&self) -> std::sync::MutexGuard<'_, Slots<T>> {
|
||
self.slots.lock().unwrap_or_else(|e| e.into_inner())
|
||
}
|
||
|
||
/// A free workspace, a new one from `make` when fewer than `cap` exist, else wait for one to come back or be
|
||
/// forgotten.
|
||
fn take(&self, make: impl FnOnce() -> Result<T, Error>) -> Result<T, Error> {
|
||
let mut s = self.lock();
|
||
loop {
|
||
if let Some(w) = s.free.pop() {
|
||
return Ok(w);
|
||
}
|
||
if s.made < self.cap {
|
||
s.made += 1;
|
||
drop(s);
|
||
let w = make();
|
||
if w.is_err() {
|
||
self.forget();
|
||
}
|
||
return w;
|
||
}
|
||
s = self.wake.wait(s).unwrap_or_else(|e| e.into_inner());
|
||
}
|
||
}
|
||
|
||
fn give(&self, w: T) {
|
||
self.lock().free.push(w);
|
||
self.wake.notify_one();
|
||
}
|
||
|
||
/// One workspace fewer exists: a waiting caller may make another.
|
||
fn forget(&self) {
|
||
let mut s = self.lock();
|
||
s.made = s.made.saturating_sub(1);
|
||
drop(s);
|
||
self.wake.notify_one();
|
||
}
|
||
}
|
||
|
||
/// A network on the GPU for one tile size. Cheap to share between threads: each call takes a free workspace.
|
||
pub struct NetRunner {
|
||
shared: Shared,
|
||
tile: usize,
|
||
in_channels: usize,
|
||
/// Workspaces are made when a call finds none free, up to [`WORKSPACES`].
|
||
workspaces: Workspaces<Workspace>,
|
||
bytes: u64,
|
||
}
|
||
|
||
/// Put `net` on the GPU for `tile × tile` input cells, or say why it cannot be (no GPU, a layer or a size the kernels
|
||
/// do not do, the device refusing the buffers). Creates the device on first use and compiles the kernels.
|
||
pub fn runner(net: &Net, tile: usize) -> Result<NetRunner, String> {
|
||
let _work = crate::exit::enter().ok_or(CLOSING)?;
|
||
let d = dev()?;
|
||
let max_binding = d.limits.max_storage_buffer_binding_size.min(d.limits.max_buffer_size).min(MAX_BUFFER);
|
||
let (layout, layers) = plan(net, tile, max_binding)?;
|
||
let scope = [
|
||
d.device.push_error_scope(wgpu::ErrorFilter::OutOfMemory),
|
||
d.device.push_error_scope(wgpu::ErrorFilter::Validation),
|
||
d.device.push_error_scope(wgpu::ErrorFilter::Internal),
|
||
];
|
||
// the driver compiles the kernels here: a crash in it is reported at the next launch (issue #250)
|
||
let pipes = crate::backend::with_marker("denoise kernel compilation", || build_pipelines(d, &layout));
|
||
let weights: Vec<(wgpu::Buffer, wgpu::Buffer)> = layers
|
||
.iter()
|
||
.map(|l| {
|
||
let make = |v: &[f32]| {
|
||
d.device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
|
||
label: Some("weights"),
|
||
contents: bytemuck::cast_slice(v),
|
||
usage: wgpu::BufferUsages::STORAGE,
|
||
})
|
||
};
|
||
(make(&l.weight), make(&l.bias))
|
||
})
|
||
.collect();
|
||
let uniforms: Vec<wgpu::Buffer> = layout
|
||
.steps
|
||
.iter()
|
||
.map(|s| {
|
||
d.device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
|
||
label: Some("params"),
|
||
contents: &s.params,
|
||
usage: wgpu::BufferUsages::UNIFORM,
|
||
})
|
||
})
|
||
.collect();
|
||
let dummy = storage(d, "dummy", 16, wgpu::BufferUsages::empty());
|
||
let bytes: u64 = layout.slots.iter().sum::<u64>() + ((layout.in_floats + layout.out_floats) * 8) as u64;
|
||
let shared = Shared { dev: d, layout, pipes, weights, uniforms, dummy };
|
||
let first = shared.workspace();
|
||
let [oom, validation, internal] = scope;
|
||
let errors = [pollster::block_on(internal.pop()), pollster::block_on(validation.pop()), pollster::block_on(oom.pop())];
|
||
if let Some(e) = errors.into_iter().flatten().next() {
|
||
return Err(format!("{}: the denoise kernels could not be set up ({e})", d.info.name));
|
||
}
|
||
Ok(NetRunner { shared, tile, in_channels: net.in_channels, workspaces: Workspaces::new(first, WORKSPACES), bytes })
|
||
}
|
||
|
||
impl NetRunner {
|
||
/// The adapter this runner uses.
|
||
pub fn adapter(&self) -> String {
|
||
format!("{} ({:?})", self.shared.dev.info.name, self.shared.dev.info.backend)
|
||
}
|
||
|
||
/// How long each kernel took to build, e.g. `conv 64x64x16: 812 ms`.
|
||
pub fn build_report(&self) -> String {
|
||
self.shared.pipes.built.iter().map(|(k, ms)| format!("{k}: {ms:.0} ms")).collect::<Vec<_>>().join(", ")
|
||
}
|
||
|
||
/// Device memory one tile in flight uses (bytes), without the weights.
|
||
pub fn bytes_per_tile(&self) -> u64 {
|
||
self.bytes
|
||
}
|
||
|
||
/// A free workspace, a new one when fewer than [`WORKSPACES`] exist, else wait for one to come back.
|
||
fn take(&self) -> Result<Workspace, Error> {
|
||
self.workspaces.take(|| {
|
||
// a caller woken by the last tiles failing on a lost device makes nothing on it
|
||
if let Some(why) = broken_reason() {
|
||
return Err(Error::Runtime(format!("the GPU is not used any more: {why}")));
|
||
}
|
||
let d = self.shared.dev;
|
||
let scopes = [d.device.push_error_scope(wgpu::ErrorFilter::OutOfMemory), d.device.push_error_scope(wgpu::ErrorFilter::Validation)];
|
||
let w = self.shared.workspace();
|
||
let [oom, validation] = scopes;
|
||
let (validation, oom) = (pollster::block_on(validation.pop()), pollster::block_on(oom.pop()));
|
||
if let Some(e) = oom.or(validation) {
|
||
return Err(Error::Runtime(format!("the GPU could not make room for another tile: {e}")));
|
||
}
|
||
Ok(w)
|
||
})
|
||
}
|
||
|
||
fn give(&self, w: Workspace) {
|
||
self.workspaces.give(w);
|
||
}
|
||
|
||
/// A workspace is gone (dropped after a failure): room for a new one, and a waiting caller may make it.
|
||
fn forget(&self) {
|
||
self.workspaces.forget();
|
||
}
|
||
|
||
fn execute(&self, w: &Workspace, input: &[f32]) -> Result<Vec<f32>, Error> {
|
||
let d = self.shared.dev;
|
||
let (t, c) = (self.tile, self.in_channels);
|
||
// planar → channels innermost
|
||
let mut nhwc = vec![0f32; input.len()];
|
||
for (ch, plane) in input.chunks_exact(t * t).enumerate() {
|
||
for (i, v) in plane.iter().enumerate() {
|
||
if let Some(o) = nhwc.get_mut(i * c + ch) {
|
||
*o = *v;
|
||
}
|
||
}
|
||
}
|
||
let mut calls = Vec::with_capacity(w.bound.len());
|
||
for (kernel, groups, bg) in &w.bound {
|
||
let pipe = match kernel {
|
||
Kernel::Conv { bm, bn, bk } => self.shared.pipes.conv.get(&(*bm, *bn, *bk)),
|
||
Kernel::Pool => Some(&self.shared.pipes.pool),
|
||
};
|
||
let Some(pipe) = pipe else { return Err(Error::Runtime("a kernel is missing".into())) };
|
||
calls.push((pipe, groups, bg));
|
||
}
|
||
let scopes = [
|
||
d.device.push_error_scope(wgpu::ErrorFilter::OutOfMemory),
|
||
d.device.push_error_scope(wgpu::ErrorFilter::Validation),
|
||
d.device.push_error_scope(wgpu::ErrorFilter::Internal),
|
||
];
|
||
d.queue.write_buffer(&w.input, 0, bytemuck::cast_slice(&nhwc));
|
||
let mut enc = d.device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: None });
|
||
{
|
||
let mut pass = enc.begin_compute_pass(&wgpu::ComputePassDescriptor { label: Some("denoise tile"), timestamp_writes: None });
|
||
for (pipe, groups, bg) in calls {
|
||
pass.set_pipeline(pipe);
|
||
pass.set_bind_group(0, bg, &[]);
|
||
pass.dispatch_workgroups(groups[0], groups[1], groups[2]);
|
||
}
|
||
}
|
||
let bytes = (self.shared.layout.out_floats * 4) as u64;
|
||
enc.copy_buffer_to_buffer(&w.output, 0, &w.staging, 0, bytes);
|
||
d.queue.submit([enc.finish()]);
|
||
let slice = w.staging.slice(..bytes);
|
||
let (tx, rx) = std::sync::mpsc::channel();
|
||
slice.map_async(wgpu::MapMode::Read, move |r| {
|
||
let _ = tx.send(r);
|
||
});
|
||
let waited = d.device.poll(wgpu::PollType::Wait { submission_index: None, timeout: Some(TIMEOUT) });
|
||
let [oom, validation, internal] = scopes;
|
||
let errors = [pollster::block_on(internal.pop()), pollster::block_on(validation.pop()), pollster::block_on(oom.pop())];
|
||
let [internal, validation, oom] = errors;
|
||
if let Some(e) = oom {
|
||
// not a broken device: this tile runs on the CPU, and the next may fit
|
||
return Err(Error::Runtime(format!("the GPU ran out of memory: {e}")));
|
||
}
|
||
if let Some(e) = internal.or(validation) {
|
||
mark_broken(&format!("device error: {e}"));
|
||
return Err(Error::Runtime(format!("the GPU reported an error: {e}")));
|
||
}
|
||
if let Err(e) = waited {
|
||
mark_broken(&format!("the GPU did not finish a tile: {e}"));
|
||
return Err(Error::Runtime(format!("the GPU did not finish: {e}")));
|
||
}
|
||
match rx.recv_timeout(Duration::from_secs(10)) {
|
||
Ok(Ok(())) => {}
|
||
Ok(Err(e)) => return Err(Error::Runtime(format!("reading the result back failed: {e}"))),
|
||
Err(e) => return Err(Error::Runtime(format!("reading the result back failed: {e}"))),
|
||
}
|
||
let out = {
|
||
let data = slice.get_mapped_range().map_err(|e| Error::Runtime(format!("reading the result back failed: {e:?}")))?;
|
||
data.as_chunks::<4>().0.iter().map(|b| f32::from_le_bytes(*b)).collect::<Vec<f32>>()
|
||
};
|
||
w.staging.unmap();
|
||
Ok(out)
|
||
}
|
||
}
|
||
|
||
impl TileRunner for NetRunner {
|
||
fn run(&self, input: &[f32]) -> Result<Vec<f32>, Error> {
|
||
if let Some(why) = broken_reason() {
|
||
return Err(Error::Runtime(format!("the GPU is not used any more: {why}")));
|
||
}
|
||
if input.len() != self.shared.layout.in_floats {
|
||
return Err(Error::Input("the tile is not the size the model wants".into()));
|
||
}
|
||
// one tile is one piece of GPU work: a photo's worth would outlast any wait at exit
|
||
let Some(_work) = crate::exit::enter() else { return Err(Error::Runtime(CLOSING.into())) };
|
||
let w = self.take()?;
|
||
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| self.execute(&w, input)));
|
||
match r {
|
||
Ok(Ok(out)) => {
|
||
self.give(w);
|
||
Ok(out)
|
||
}
|
||
Ok(Err(e)) => {
|
||
// its readback buffer may be left half-mapped: a fresh workspace is made when one is next needed
|
||
drop(w);
|
||
self.forget();
|
||
Err(e)
|
||
}
|
||
Err(_) => {
|
||
// a panic inside wgpu: this workspace is not given back, and the GPU is not used again
|
||
mark_broken("a denoise tile panicked");
|
||
drop(w);
|
||
self.forget();
|
||
Err(Error::Runtime("the GPU runner gave up".into()))
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
/// The denoise kernels validate without a GPU: the pooling kernel, and the convolution for every block size the
|
||
/// planner picks for these layer widths and channel counts.
|
||
#[test]
|
||
fn the_network_kernels_validate_without_a_gpu() {
|
||
assert_eq!(crate::validate_wgsl("nn_pool", include_str!("wgsl/nn_pool.wgsl")), Ok(()));
|
||
let mut sizes: Vec<(u32, u32, u32)> = Vec::new();
|
||
for n in [1usize, 3, 8, 16, 24, 32, 48, 64, 96, 128, 256, 512] {
|
||
for cin in [1usize, 3, 4, 8, 16, 32, 48, 64, 128, 256, 512] {
|
||
let b = blocks(n, cin);
|
||
if !sizes.contains(&b) {
|
||
sizes.push(b);
|
||
}
|
||
}
|
||
}
|
||
for (bm, bn, bk) in sizes {
|
||
assert_eq!(crate::validate_wgsl(&format!("nn_conv {bm}x{bn}x{bk}"), &conv_source(bm, bn, bk)), Ok(()));
|
||
}
|
||
}
|
||
use lightcraft_denoise::{onnx, reference, synthetic};
|
||
|
||
fn net(tile: u64, ch: u64, depth: usize, seed: u64) -> Net {
|
||
let path = std::env::temp_dir().join(format!("lc-nn-{}-{tile}-{ch}-{depth}-{seed}.onnx", std::process::id()));
|
||
std::fs::write(&path, synthetic::unet_onnx(tile, ch, depth, seed)).unwrap();
|
||
let n = onnx::read(&path).unwrap();
|
||
let _ = std::fs::remove_file(&path);
|
||
n
|
||
}
|
||
|
||
fn picture(len: usize, seed: u64) -> Vec<f32> {
|
||
let mut s = seed | 1;
|
||
(0..len)
|
||
.map(|_| {
|
||
s ^= s >> 12;
|
||
s ^= s << 25;
|
||
s ^= s >> 27;
|
||
(s.wrapping_mul(0x2545_f491_4f6c_dd1d) >> 40) as f32 / (1u64 << 24) as f32
|
||
})
|
||
.collect()
|
||
}
|
||
|
||
/// The runner, or `None` (with a note) on a machine with no usable GPU.
|
||
fn gpu(net: &Net, tile: usize) -> Option<NetRunner> {
|
||
// creating the device and building the kernels arms the crash sentinel: take turns with
|
||
// the tests that set its path (`backend::tests`)
|
||
let _serial = crate::DEVICE_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner());
|
||
match runner(net, tile) {
|
||
Ok(r) => Some(r),
|
||
Err(e) => {
|
||
lightcraft_testkit::skip!(Gpu, "no GPU runner here ({e})");
|
||
None
|
||
}
|
||
}
|
||
}
|
||
|
||
fn worst(a: &[f32], b: &[f32]) -> f32 {
|
||
assert_eq!(a.len(), b.len());
|
||
assert!(a.iter().chain(b).all(|v| v.is_finite()), "non-finite denoise output");
|
||
let scale = b.iter().fold(1e-3f32, |m, v| m.max(v.abs()));
|
||
a.iter().zip(b).fold(0f32, |m, (x, y)| m.max((x - y).abs())) / scale
|
||
}
|
||
|
||
/// A 1 × 1 convolution from 4 channels to `cout` that copies input channel `co % 4` to each output channel `co`.
|
||
fn copy(src: usize, cout: usize) -> Op {
|
||
let mut weight = vec![0f32; cout * 4];
|
||
for co in 0..cout {
|
||
weight[co * 4 + co % 4] = 1.0;
|
||
}
|
||
Op::Conv(lightcraft_denoise::net::Conv { src, src2: None, cin: 4, cout, k: 1, weight, bias: vec![0.0; cout], leaky: None })
|
||
}
|
||
|
||
/// Two 1 × 1 layers that only copy channels, so the right answer is exact: the first layer's stores (mode 0) went
|
||
/// wrong on DX12 with FXC on an NVIDIA GTX 1070, where every second pixel came out with its channels shifted.
|
||
#[test]
|
||
fn a_layer_stores_every_pixels_channels_where_the_next_reads_them() {
|
||
let n = Net::new(4, vec![copy(0, 4), copy(1, 12), Op::DepthToSpace2 { src: 2 }]).unwrap();
|
||
let Some(g) = gpu(&n, 3) else { return };
|
||
let x: Vec<f32> = (1..=36).map(|i| i as f32).collect();
|
||
assert_eq!(g.run(&x).unwrap(), reference::run(&n, 3, &x).unwrap());
|
||
}
|
||
|
||
/// The same for a transposed convolution's stores (mode 1): a max-pool halves the tile, and the 2 × 2 transposed
|
||
/// convolution copies each pooled pixel's channels to the four pixels it becomes.
|
||
#[test]
|
||
fn a_transposed_convolution_stores_every_pixels_channels_where_the_next_reads_them() {
|
||
use lightcraft_denoise::net::ConvT2;
|
||
// [cin][cout][2][2]: channel c to channel c at all four places
|
||
let mut weight = vec![0f32; 4 * 4 * 4];
|
||
for c in 0..4 {
|
||
for q in 0..4 {
|
||
weight[(c * 4 + c) * 4 + q] = 1.0;
|
||
}
|
||
}
|
||
let up = Op::ConvT2(ConvT2 { src: 1, cin: 4, cout: 4, weight, bias: vec![0.0; 4], leaky: None });
|
||
let n = Net::new(4, vec![Op::MaxPool2 { src: 0 }, up, copy(2, 12), Op::DepthToSpace2 { src: 3 }]).unwrap();
|
||
let Some(g) = gpu(&n, 6) else { return };
|
||
let x: Vec<f32> = (1..=144).map(|i| i as f32).collect();
|
||
assert_eq!(g.run(&x).unwrap(), reference::run(&n, 6, &x).unwrap());
|
||
}
|
||
|
||
#[test]
|
||
fn the_gpu_gives_what_the_reference_gives() {
|
||
// (tile, channels, depth): a tile that is not a multiple of the block size, odd channel counts, a deeper net
|
||
for (tile, ch, depth, seed) in
|
||
[(1u64, 4u64, 0usize, 1u64), (3, 4, 0, 8), (6, 4, 1, 9), (20, 12, 2, 2), (32, 8, 2, 3), (48, 12, 2, 4), (64, 16, 3, 5)]
|
||
{
|
||
let n = net(tile, ch, depth, seed);
|
||
let Some(g) = gpu(&n, tile as usize) else { return };
|
||
let x = picture(4 * (tile * tile) as usize, seed);
|
||
let want = reference::run(&n, tile as usize, &x).unwrap();
|
||
let got = g.run(&x).unwrap();
|
||
let cpu = lightcraft_denoise::cpu::NetRunner::new(&n, tile as usize).unwrap().run(&x).unwrap();
|
||
assert!(worst(&got, &cpu) < 1e-3, "GPU and pure-Rust CPU differ");
|
||
let e = worst(&got, &want);
|
||
eprintln!("{}: tile {tile}, {ch} channels, depth {depth}: off by {e:e} of the output's size", g.adapter());
|
||
assert!(e < 2e-4, "tile {tile}, {ch} channels, depth {depth}: the GPU is off by {e:e} of the output's size");
|
||
}
|
||
}
|
||
|
||
// Issue #479: several pixels loaded by different invocations shared a vec4,
|
||
// so lane writes raced on Metal. Exercise a partial block repeatedly with new
|
||
// inputs; stale shared-memory values must never leak into the next dispatch.
|
||
#[test]
|
||
fn partial_tiles_remain_correct_when_reusing_a_runner() {
|
||
let n = net(3, 4, 0, 8);
|
||
let Some(g) = gpu(&n, 3) else { return };
|
||
for seed in 100..112 {
|
||
let x = picture(4 * 3 * 3, seed);
|
||
let want = reference::run(&n, 3, &x).unwrap();
|
||
let e = worst(&g.run(&x).unwrap(), &want);
|
||
assert!(e < 2e-4, "seed {seed}: off by {e:e}");
|
||
}
|
||
}
|
||
|
||
#[test]
|
||
fn a_runner_serves_many_tiles_from_many_threads() {
|
||
let n = net(32, 8, 2, 7);
|
||
let Some(g) = gpu(&n, 32) else { return };
|
||
let inputs: Vec<Vec<f32>> = (0..12).map(|i| picture(4 * 32 * 32, 100 + i)).collect();
|
||
let wants: Vec<Vec<f32>> = inputs.iter().map(|x| reference::run(&n, 32, x).unwrap()).collect();
|
||
std::thread::scope(|s| {
|
||
for chunk in inputs.chunks(3).zip(wants.chunks(3)) {
|
||
let g = &g;
|
||
s.spawn(move || {
|
||
for (x, want) in chunk.0.iter().zip(chunk.1) {
|
||
let e = worst(&g.run(x).unwrap(), want);
|
||
assert!(e < 2e-4, "off by {e:e}");
|
||
}
|
||
});
|
||
}
|
||
});
|
||
}
|
||
|
||
/// The real model on the GPU against the pure-Rust CPU runner, with timings (the numbers in docs/denoise.md come from here):
|
||
/// `LC_DENOISE_MODEL=<model_bayer.onnx> cargo test --release -p lightcraft-gpu --lib real_model -- --ignored --nocapture`
|
||
#[test]
|
||
#[ignore = "needs the real model: see the doc comment"]
|
||
fn real_model_on_the_gpu_matches_cpu() {
|
||
use lightcraft_denoise::manifest::{DenoiserManifest, Domain, Gain};
|
||
use lightcraft_denoise::runtime::CpuRunner;
|
||
use std::time::Instant;
|
||
let Some(model) = std::env::var_os("LC_DENOISE_MODEL") else { return };
|
||
let net = onnx::read(std::path::Path::new(&model)).unwrap();
|
||
let tile = 512usize;
|
||
let started = Instant::now();
|
||
let g = {
|
||
let _serial = crate::DEVICE_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner());
|
||
runner(&net, tile).unwrap()
|
||
};
|
||
println!("{}: set up in {:.2} s, {:.0} MB per tile in flight", g.adapter(), started.elapsed().as_secs_f64(), g.bytes_per_tile() as f64 / 1e6);
|
||
println!("kernels: {}", g.build_report());
|
||
let manifest = DenoiserManifest {
|
||
id: "rawnind-bayer".into(),
|
||
name: "RawNIND Bayer".into(),
|
||
version: "1".into(),
|
||
licence: Default::default(),
|
||
source: None,
|
||
sha256: None,
|
||
size_bytes: None,
|
||
provenance: String::new(),
|
||
domain: Domain::BayerToRgb,
|
||
tile: 512,
|
||
overlap: 64,
|
||
gain: Gain::MatchMean { nominal: 1.0e6, max_deviation: 0.05 },
|
||
};
|
||
let cpu = CpuRunner::load(std::path::Path::new(&model), &manifest).unwrap();
|
||
// a gradient with noise, in the model's input range
|
||
let noise = picture(4 * tile * tile, 42);
|
||
let input: Vec<f32> =
|
||
(0..4 * tile * tile).map(|i| (0.2 + 0.3 * ((i % tile) as f32 / tile as f32) + 0.1 * (noise[i] - 0.5)).clamp(0.0, 1.0)).collect();
|
||
let t = Instant::now();
|
||
let want = cpu.run(&input).unwrap();
|
||
println!("CPU: {:.0} ms for one tile (one call runs on one core)", t.elapsed().as_secs_f64() * 1000.0);
|
||
let t = Instant::now();
|
||
let got = g.run(&input).unwrap();
|
||
println!("gpu, first tile: {:.0} ms", t.elapsed().as_secs_f64() * 1000.0);
|
||
let e = worst(&got, &want);
|
||
println!("worst difference {e:e} of the output's size");
|
||
assert!(e < 1e-3, "the GPU does not give the CPU answer ({e:e})");
|
||
let mut times: Vec<f64> = (0..8)
|
||
.map(|_| {
|
||
let t = Instant::now();
|
||
g.run(&input).unwrap();
|
||
t.elapsed().as_secs_f64() * 1000.0
|
||
})
|
||
.collect();
|
||
times.sort_by(|a, b| a.total_cmp(b));
|
||
println!("gpu, one tile at a time: median {:.0} ms (best {:.0}, worst {:.0})", times[times.len() / 2], times[0], times[times.len() - 1]);
|
||
// two threads, as the denoiser feeds it
|
||
let t = Instant::now();
|
||
std::thread::scope(|s| {
|
||
for _ in 0..2 {
|
||
s.spawn(|| {
|
||
for _ in 0..8 {
|
||
g.run(&input).unwrap();
|
||
}
|
||
});
|
||
}
|
||
});
|
||
println!("gpu, 16 tiles from 2 threads: {:.0} ms per tile", t.elapsed().as_secs_f64() * 1000.0 / 16.0);
|
||
}
|
||
|
||
/// How long the driver takes to build one conv kernel variant: `LC_SHADER=<file.wgsl> LC_BLOCKS=128,32,16`.
|
||
#[test]
|
||
#[ignore = "a probe for shader compile times"]
|
||
fn time_a_shader_build() {
|
||
let Some(path) = std::env::var_os("LC_SHADER") else { return };
|
||
let blocks: Vec<u32> = std::env::var("LC_BLOCKS").unwrap_or_else(|_| "128,32,16".into()).split(',').filter_map(|v| v.parse().ok()).collect();
|
||
let (bm, bn, bk) = (blocks[0], blocks[1], blocks[2]);
|
||
let src = std::fs::read_to_string(path)
|
||
.unwrap()
|
||
.replace("{{BM}}", &bm.to_string())
|
||
.replace("{{BN}}", &bn.to_string())
|
||
.replace("{{BK}}", &bk.to_string())
|
||
.replace("{{B_VECS}}", &(bk * bn / 4).to_string());
|
||
let d = dev().unwrap();
|
||
let layout = conv_bgl(d);
|
||
let started = std::time::Instant::now();
|
||
let _ = pipeline(d, "probe", &src, &layout);
|
||
println!("{} {bm}x{bn}x{bk}: built in {:.0} ms", d.info.backend.to_str(), started.elapsed().as_secs_f64() * 1000.0);
|
||
}
|
||
|
||
#[test]
|
||
fn a_tile_of_the_wrong_size_is_an_error_not_a_crash() {
|
||
let n = net(16, 4, 1, 9);
|
||
let Some(g) = gpu(&n, 16) else { return };
|
||
assert!(g.run(&[0.0; 10]).is_err());
|
||
assert!(g.run(&[]).is_err());
|
||
}
|
||
|
||
#[test]
|
||
fn networks_the_kernels_cannot_run_are_refused() {
|
||
let n = net(16, 4, 1, 11);
|
||
// a tile the network does not fit
|
||
assert!(plan(&n, 15, u64::MAX).is_err());
|
||
// a buffer limit smaller than the activations
|
||
assert!(plan(&n, 16, 64).is_err());
|
||
}
|
||
|
||
/// Issue #2577: a tile failing between a caller finding every workspace in use and that caller waiting still wakes
|
||
/// it. The caller holds the lock from its check to its wait, as `Workspaces::take` does, so `forget` waits for it. A
|
||
/// wait that runs out fails, whatever the count is by then: the wakeup itself has to arrive.
|
||
#[test]
|
||
fn a_caller_that_finds_every_workspace_in_use_hears_one_being_forgotten() {
|
||
let ws = Workspaces::new(0u32, 1);
|
||
assert_eq!(ws.lock().free.pop(), Some(0));
|
||
std::thread::scope(|scope| {
|
||
// where take() is between finding every workspace in use and waiting
|
||
let mut s = ws.lock();
|
||
assert!(s.free.is_empty() && s.made == ws.cap);
|
||
scope.spawn(|| ws.forget());
|
||
// the tile in flight fails now
|
||
std::thread::sleep(Duration::from_millis(50));
|
||
// wait as take() does, and again only after a spurious wakeup that changed nothing
|
||
loop {
|
||
let (next, r) = ws.wake.wait_timeout(s, Duration::from_secs(10)).unwrap_or_else(|e| e.into_inner());
|
||
assert!(!r.timed_out(), "the caller missed the workspace being forgotten");
|
||
s = next;
|
||
if s.made != ws.cap {
|
||
break;
|
||
}
|
||
}
|
||
assert_eq!(s.made, 0);
|
||
});
|
||
}
|
||
|
||
#[test]
|
||
fn take_waits_for_a_workspace_to_come_back_or_be_forgotten() {
|
||
let ws = std::sync::Arc::new(Workspaces::new(0u32, 1));
|
||
let fail = || Err(Error::Runtime("no room".into()));
|
||
assert_eq!(ws.take(fail).unwrap(), 0);
|
||
// a caller on a thread nobody joins: if its wakeup never comes, the test fails on its timeout instead of hanging
|
||
let caller = |made: u32| {
|
||
let (ws, (tx, rx)) = (ws.clone(), std::sync::mpsc::channel());
|
||
std::thread::spawn(move || tx.send(ws.take(|| Ok(made))));
|
||
rx
|
||
};
|
||
// the only workspace is in flight: a caller waits until it is forgotten, then makes a new one
|
||
let rx = caller(1);
|
||
assert!(rx.recv_timeout(Duration::from_millis(50)).is_err(), "took a workspace while none was free");
|
||
ws.forget();
|
||
assert_eq!(rx.recv_timeout(Duration::from_secs(10)).unwrap().unwrap(), 1);
|
||
// or until it comes back, and takes that one
|
||
let rx = caller(2);
|
||
assert!(rx.recv_timeout(Duration::from_millis(50)).is_err(), "took a workspace while none was free");
|
||
ws.give(1);
|
||
assert_eq!(rx.recv_timeout(Duration::from_secs(10)).unwrap().unwrap(), 1);
|
||
// a workspace that could not be made leaves room for the next try
|
||
ws.forget();
|
||
assert!(ws.take(fail).is_err());
|
||
assert_eq!(ws.lock().made, 0);
|
||
assert_eq!(ws.take(|| Ok(3)).unwrap(), 3);
|
||
assert_eq!(ws.lock().made, 1);
|
||
}
|
||
}
|