* MIDI input devices: live play-through and MIDI recording Keyboards and pads could not be used at all: there was no MIDI device layer (G-02). - crates/playback/src/midi.rs opens every MIDI input port through midir (ALSA sequencer, CoreMIDI, WinMM), follows devices that are plugged in or removed, and passes channel messages to the audio thread over a lock-free ring, each with the time it arrived. - The mix engine plays them on the instruments of record-enabled and input-monitored MIDI and instrument tracks (crates/mix/src/live_midi.rs): notes at their sample offset, the sustain pedal holding note-offs back. - While recording, the events are kept with their timeline position and become MIDI clips on the record-enabled tracks (crates/engine/src/io/midi_take.rs), with controllers, pitch bend and aftertouch. A pass can be MIDI only: the audio input is opened for audio tracks alone. - A MIDI or instrument track chooses a device and a channel at its input selector (kept in the track's input route as `MIDI:<channel>:<device>`). - Setup > Playback Engine lists the input ports with a switch each and, on request, what arrives from one; Setup > MIDI > MIDI Input Devices opens it. The switches are app preferences. Note lifecycle: - A strip keeps the keys and the sustain pedal of every owner (input port and channel) apart. A pitch held by two devices or channels sounds until the last one releases it, and one owner's pedal does not hold another's notes. A strip follows 16 owners at a time; further ones share the last place (documented in live_midi.rs). - A full queue is never silent. An event that does not reach the audio thread (the 1,024-entry input ring or the 256 events a callback takes) marks its port; the audio thread then drops what it still has queued from that port and the engine releases the notes and pedal the port holds. A port that closes, a replaced feed and events skipped while nothing listens take the same path. - A take records where input was dropped: the notes that port held end there, and the status line says the take is incomplete. Recorded events the UI thread did not collect in time are counted and reported too. Recorded notes are told apart by device as well as channel and pitch. On Linux, live play was checked with a hardware keyboard (Akai MPK mini 3) and recording with a virtual one (crates/playback/examples/midi_keys.rs), before the note-lifecycle work; that part is covered by tests only. macOS and Windows are type-checked only; there is no MIDI input on FreeBSD or the web. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * midi real device tests * midi device drop test * midi device fix pr review comments * MIDI input: review fixes for note ownership, lost input and takes Live notes (crates/mix/src/live_midi.rs): - An owner without a place of its own joins the shared place while it is in use, even when another place has come free. It could get a second place before, and the release of its key in the shared place was then not found: the note stayed on. The same held for the pedal. - A change of the track's input releases the shared place when the input no longer takes any one of the owners in it, not only the first. - The strip remembers which port and channel last set each controller. When that source loses its input or the track stops listening to it, the instrument's controllers are set again from the clips and from the sources that are left, so a bend from an unplugged device does not stay. - When playback runs out under monitoring, the clips' sustain pedal goes up with their notes. Capture (crates/playback): - A loss carries its time. What a port sends after it is played and comes after the mark in the take; before, it was dropped and recorded ahead of the mark. - The age of an event is counted at the transport speed. - Loop recording writes a mark where the loop goes round, and a key played with the playhead at the end of the loop stays inside the loop. - The end of a record pass is acknowledged by its number, so the late answer to an earlier pass does not end the wait for a later one. - A connection the backend closes marks its port, also when a device with the same name is opened in its place. Takes (crates/engine/src/io/midi_take.rs, crates/ui-egui): - A note held into the next round of a loop is split there, also when its release is later on the timeline than its press. - A track taken out of record during a pass keeps what it recorded. - A pass whose end the audio engine did not confirm is reported as incomplete in the status line. - An inactive record-enabled audio track no longer turns a MIDI pass into an audio one that is then refused. docs/midi-input.md: Windows drivers that allow one program per device, and the new status text. * MIDI input: review follow-ups on loss order, controllers and loop takes - Each port has its own time of loss. One port's later loss no longer takes along what another port sent after its own: that is played, and comes after that port's mark in the take. - A retired connection is closed before its port is marked as lost. Marked first, a note-on delivered while it closed carried a later time than the loss and could be played after the release. - The strip keeps the last controller value of every source, not only the newest. When one of two sources that set the same controller goes away, the controller goes back to the value of the one that stays. 64 values are kept; past that the oldest is forgotten. - A take can say where the pass stopped (MidiTakeEvent::END), and the UI says so for loop passes. A key still down then ends at the stop in the round it is in: held over a round or more it covers the loop, where it used to end at the loop end of its first round. - A pass that records MIDI does not start while the audio engine has not confirmed the end of the previous one. It takes the MIDI since its last callback with the state of the pass it knows, so a pass started and stopped meanwhile had its notes written at the old position. * MIDI input: loss marks in the queue, connection gate, stop position - A loss from a full queue is also written into the queue, ahead of the first event that fits again. The audio thread deals with it at that place. Before, a loss flagged after the callback had read the flags was seen one callback late and ended a note played after it. The flag for a loss already met in the queue is not a second loss. - A connection is shut off under the feed's lock when it is closed (MidiFeed::close_port), and what its device thread still delivers is discarded. Dropping a connection joins that thread on ALSA; CoreMIDI (MIDIPortDispose) and WinMM (midiInStop, midiInReset, midiInClose) do not promise to, so the order of drop and mark is no longer relied on. - When the controller table is full and forgets a value, the strip keeps the port and channel it came from. The controllers are still reset when that source goes away, where a bend pushed out of the table used to stay. - A loop pass ends where the audio engine is. The larger of the engine's and the UI's position was used, and just after the loop went round the UI's is still near the loop end: a short held note covered the loop. - The end of a pass waits for the audio engine by what the pass set out to record, not by the devices connected at Stop. A pass whose last device was unplugged before Stop left nothing pending, and a new pass could start before the engine was done with it. * MIDI input: stop at the loop end, and passes owed by the audio engine - A key played while the playhead stands at the end of the loop is the last thing of that round. It was written as the first of the next round, with a mark that the loop had gone round; a pass stopped right there never went round, and a note held to the stop grew a second part. - The feed knows whether an ended pass is still owed by the audio engine (MidiFeed::pass_owed), and the UI asks it before a pass that records MIDI starts. The UI kept that itself, where no test could stall the audio callback against it. * MIDI input: stop boundary, loop rounds, loss marks and shared owners - A pass ends at a time on the feed's clock (MidiFeed::end_pass). Only what arrived up to it is written to the take, however late the audio thread hears of the Stop, and the callback takes the queue in once more after the command before it confirms: a note that arrived between its reading the queue and the Stop was missing from a confirmed take. - The confirmation carries where the playhead stood (MidiFeed::stopped_at). A loop pass ends there; the player's position read afterwards could be a round further on, which the take knows nothing about. Without a confirmation no stop position is used. - The take builder is told whether the pass looped (MidiPass). In a pass that did not, a release stamped before its key press (a late callback) is one short note, where it was split around the ends of the clip. The stop position is a parameter, not an entry that the event cap could cut. - A pass records the tracks that were in record when it started, and only those: the same list the end of the pass waits by. A track put in record during the pass got what was played before that, without a wait. - A short loop goes round several times in a callback: each entry is put in the round it arrived in, so two entries a round apart have one mark between them. - A mark in the queue is stamped with the time of its loss. A port that overflowed, played on and was then closed keeps a mark for the close (the new note was held to the end of the take), and a loss told by the port's flag is not written again when its mark arrives. - Live events stay in order of arrival when they are played: a release taken by a shorter callback was due before its note-on. - A callback reads the transport speed once, for rendering and for the ages of the next callback's events. - The shared owner place is left alone by a source that never joined it: its note-off no longer ends another owner's note there, and its loss no longer releases the place. - When a live source goes away, the instrument's controllers are restored before the block's clip events are sent, so a clip controller at the start of that block keeps its new value. - Tests that assert MIDI positions use a clock the test sets. * MIDI input: pass boundaries, loop punch-in and review follow-ups - A pass begun while the transport runs has a start time on the feed (MidiFeed::begin_pass): what is played after Record is written down also when the audio thread takes it in before it reads the command. - A pass goes on through a transport stop until the command that ends it. The UI sends the two one after the other; when they reached two callbacks, a note played just before the stop was left out of a confirmed take. - The end of a pass is asked for every entry, so a Stop that comes while the queue is being read still cuts off what follows it. - A mark in the queue carries the time its events were dropped, kept apart from a later close of the port. An overflow before Stop is in the take when the port closes after it. - A pass punched in while a loop plays is a loop take over that loop. In a line from the punch-in to a stop in the next round nothing fell inside the range. - In a loop take with marks, only the marks say that the loop went round: a release stamped before its press within one round is one short note. - A key pressed in the last samples of the range starts at the last tick the clip plays, not at the tick of its end. - When the clips' controllers are sent again while stopped, their sustain pedal goes up again: no pedal-up of theirs would follow. - The render step is taken after the commands, so a session at another sample rate is rendered at that rate from the callback that installs it. - The stop position and the incomplete-take warning go by one answer from the audio engine. * MIDI input: test that a callback renders at the speed it keeps A speed change that arrives while a callback runs takes effect from the next one: the callback renders at the speed it read, and the next callback counts the ages of its events at that same speed. * MIDI input: volume and pan go back to MIDI's defaults with their device The reset sent when a live source goes away is Reset All Controllers, which leaves the channel volume (CC 7) and the pan (CC 10) alone. A device that turned the volume down to nothing and was unplugged left the track silent. When the source that went away had set one of the two, and neither the clips nor a source that is still there gives it a value, the instrument is sent the value MIDI names as the default (100, and 64). The same when the track stops playing live input. * MIDI input: third round of review follow-ups - The stop cutoff is one value on the feed, in force from the store that ends the pass and gone when the audio thread confirms it. - The start of a pass holds for the whole pass: what was played before Record stays out also when it is taken in after the start command. - What is played just before a Stop is put inside the range that ends at the stop, not on its end. - A port closed while the queue is read is met through the mark of its overflow; the flag of that close, a callback later, is not a second loss (it ended a key the new connection played). - An overflow and the loss it is have one time. Read apart, they could differ by a microsecond and count as two losses. - A volume or pan the full controller table forgot goes back to its default when its own device goes, not when another does. - The chase of the clips' controllers uses the track's MIDI delay, as playing them does. - Recorded controllers in the last samples of the range stay where the clip plays; in a take without marks the press as played, not the note's start, says whether the loop went round; a take cut at the event cap does not use the stop position. - The loop the MIDI of a punched-in pass goes to is the loop the audio engine plays: not a range too short to loop, nor one that Play From dropped. - The audio input stops taking in before the wait for the MIDI. * MIDI input: fourth round of review follow-ups - A record pass goes on through the natural end of playback until the UI ends it, up to the time playback ran out: what was played as it reached the end is in the take, what follows is not. - Before the end of a pass is confirmed, the device threads are kept out of the queue while it is read once more, so a message stamped before the Stop is not put in after the answer. The audio thread does not wait for a device thread that is in there. - A later loss of a port is noted as dealt with only at the end of the callback: with two overflow marks of one port in the queue, the second was skipped and the key played between them had no end. - Whether a pass was begun is read before when: a pass seen as begun has its own start time. - The volume and the pan go back to their defaults when the source that set them last goes, also when the controller table forgot its value, and not when an earlier setter goes. - A clip's note-off that is held back because a live key holds the pitch is remembered for the block: the key going up earlier in the block no longer ends the note before the clip would have. - In a take without marks, whether the loop went round between a press and its release is judged by their samples, not their ticks. - The stop position of a loop pass is not used when recorded entries did not reach the take: one may be the mark of a round. - Loop recording over a selection too short to loop is recorded as a punch, as the transport plays it. * MIDI input: fifth round of review follow-ups - Whether a pass was begun is asked for every event: a pass begun while the callback is reading the queue records from when it began. - The ages of a pass's last events are counted up to when the transport stopped. Counted up to the callback that takes them in, the time the playhead stood still moved them back. - An overflow the port's flag told of keeps its time when an older mark in the queue shows a later close: its own mark may never come. - What becomes of the clips' held back note-offs is settled after the block's live events. A key that goes down again before the clip's note would have ended is not ended by it, and with two clip notes of a pitch in a block the one that sounds when the key goes up lasts to its end. - When the source that set the volume or the pan last goes and another one's value is sent again, that one is the last setter from then on. * MIDI input: sixth round of review follow-ups - A loss is followed whenever the transport runs, and written to the take by when it happened, like an event: a port lost in a pass that began while the queue was being read ends its notes in the take. - A pass started and ended by the commands of one callback is still that pass in the last read of the queue: what was played in it is kept. - When a live key takes a pitch again, the clips' held back note-offs from since the last key went up are sent where they were, so the instrument lets go of the clip's note before the new press. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
12 KiB
Control protocol
SoundCraft can be driven completely by other programs: scripts, test harnesses and AI agents. Every menu item, button, mouse gesture and keystroke maps to a command (or to synthetic input), and every command is reachable from:
| Surface | How |
|---|---|
| Desktop app control channel | soundcraft --control <port> (0 = pick a free port; printed as SOUNDCRAFT_CONTROL_PORT=…) |
| CLI against a running app | soundcraft-cli app --port <port> <method or command id> [JSON] (see cli.md) |
| Headless CLI | soundcraft-cli run --demo --cmd 'mix.volume={"track":"Kick","db":-6}' --bounce out.wav |
| MCP (headless) | soundcraft-cli mcp [--demo] |
| MCP (bridged to the app) | soundcraft-cli mcp --connect <port> |
Isolated automation profiles
Set SOUNDCRAFT_CONFIG_DIR to a fresh directory to isolate UI preferences, Autosave, Presets, and logs.
The nonempty value is used as the complete settings directory, even in portable mode (portable.txt beside the executable, which otherwise uses SoundCraftData next to it). Unset retains the normal platform paths;
an explicit empty value disables the settings directory instead of falling back to a personal profile. Save synthetic sessions to explicit scratch paths. Do not use SOUNDCRAFT_NO_PREFS
when testing preference persistence across restarts.
Run UI acceptance with --no-audio. This disables the output player; it does not disable recording
input opened by arm/monitor commands. An automated UI smoke run must never arm a track, enable input
monitoring, or start recording. Control input injects egui events; it does not establish operating-system
input delivery or screen-reader acceptance. Native screenshots use the presented viewport frame.
Wire format
Newline-delimited JSON over TCP on 127.0.0.1. One request per line, one reply per line:
{"id": 1, "method": "engine.execute", "params": {"command": "track.new", "params": {"count": 2, "format": "Stereo"}}}
{"id": 1, "ok": true, "result": {"tracks": [12, 13]}}
Errors: {"id": 1, "ok": false, "error": "bad parameters for mix.volume: no track named Nope"}. A line that is not a JSON object with a string method gets one error reply, and the connection is closed.
Methods
| Method | Params | What it does |
|---|---|---|
engine.execute (alias command) |
{command, params} |
Run a command programmatically. Never opens a dialog; empty params use defaults or the current selection. |
engine.commands |
{filter?} |
Every command: id, label, menu path, shortcut, params doc, enabled + reason. Includes UI-layer commands (window.*). |
engine.parity |
{} |
Parity report against the incumbent's menu catalog. |
session.inspect |
`{detail?: "summary" | "full"}` |
ui.inspect |
{} |
UI state (window, panels, dialogs), window size, playing/position, audio device, MIDI inputs (midi_inputs: name, enabled, open; midi_events: messages received so far; midi_events_dropped: messages a full input queue dropped), and edit_layout (timeline rect and every track row's rect, for clicking). |
ui.set |
any UiState fields |
e.g. {"window": "Mix", "narrow_mix": true}. |
ui.menu.list |
{} |
Every catalog menu path and the command that implements it (or null). |
ui.menu.invoke |
{path} or {id} |
Like clicking the menu item: may open its dialog (e.g. "Track > New..."). |
ui.click |
{x, y, button?, count?, cmd?, shift?, alt?, ctrl?} |
Synthetic click (window points). The reply is sent only after a UI pass has applied the press/release. |
ui.drag |
{x, y, to_x, to_y, steps?} |
Synthetic drag with the primary button. |
ui.scroll |
{x, y, dx?, dy?, unit?: "point"|"line", steps?, cmd?, shift?, alt?, ctrl?} |
A two-finger trackpad swipe over (x, y): dx/dy points spread over steps frames (default 8). Positive values move the content right/down (natural scrolling). unit: "line" sends mouse-wheel notches instead. With cmd it zooms; with shift it scrolls sideways. Point scrolling is trackpad-like: egui 0.37 scroll areas may rubber-band at an edge and spring back after the gesture, so a script that needs an exact final offset should use unit: "line" or drive the target with a command. |
ui.zoom |
{x, y, factor, steps?} |
A trackpad pinch around (x, y): factor > 1 spreads the fingers (zooms in). |
ui.move |
{x, y} |
Move the pointer. |
ui.key |
{key, cmd?, shift?, alt?, ctrl?} |
Key press (Space, Enter, A, F7, =…). |
ui.text |
{text} |
Type text into the focused field. At most 64 KiB of UTF-8 text per request. |
ui.screenshot |
{path?} |
PNG of the window (base64 in png_base64 when no path). Exactly one reply per request. A request is bound to the window geometry, focus state and open session it was admitted for. A geometry or DPI change is retried within its 8 s deadline, then fails screenshot stale; a change of focused window or of the open document cancels it (screenshot canceled: focus changed / ... the session changed) because those pixels must not be reported as a capture of the new state. It can also fail screenshot queue full (8 pending, 4 in flight), screenshot too large (64 MiB per image, 192 MiB aggregate encoding peak), screenshot path longer than 4096 bytes, or screenshot unavailable: the window has no capturable size yet (a first request before any window geometry is known, e.g. minimised before the first paint). Each request reserves its own worst-case encode peak — the captured RGBA image plus the raw copy, PNG buffer and base64 text built during encoding — so several small captures cannot retain or encode full-size images past the aggregate. The callback never writes a file: encoding and writing happen after collection, and a result arriving after the deadline returns screenshot timed out without creating the file. A native hidden/minimised window is asked for an explicit paint; browser hidden tabs cannot paint on request and time out. |
app.quit |
{} |
Quit. |
Scripted input (ui.click, ui.drag, ui.scroll, ui.zoom, ui.key, ui.text, ui.move) queues at most 4096 waiting events; a request that would exceed that bound fails with synthetic input queue full instead of growing the queue. A gesture's reply is delivered only after a UI pass has applied all of its events. Requests that follow it on the same connection (inspect, engine commands, screenshots) are therefore ordered after the gesture; requests already queued behind an undelivered gesture wait as well. If no UI pass can deliver the gesture within 15 s, it is canceled with synthetic input was not delivered and the request chain continues. Scripted input is automation, not a clipboard action: copy/selection commands it triggers are not published to the OS clipboard or PRIMARY selection.
Positions and references
- Tracks: id (number) or name (
"Kick"). - Clips: id (from
session.inspect). - Positions: samples (
96000), seconds ({"seconds": 2.5}), or a time string in any timebase, optionally prefixed:"bars_beats:5|1|000","min_secs:0:12.500","timecode:00:00:10:00","samples:48000". - Ranges:
start+end(orstart+length); omitted = the current edit selection.
Examples
P=$(soundcraft --demo --control 0 | sed -n 's/SOUNDCRAFT_CONTROL_PORT=//p') # or read stderr
soundcraft-cli app --port $P mix.solo '{"track":"Bass"}'
soundcraft-cli app --port $P transport.play
soundcraft-cli app --port $P ui.screenshot '{"path":"/tmp/shot.png"}'
soundcraft-cli app --port $P ui.menu.invoke '{"path":"Track > New..."}'
Offscreen (no window): cargo run -p soundcraft-ui-egui --example ui_shot -- out.png script.jsonl,
where each script line is a request ({"method": …, "params": …}), {"shot": "path.png"} or
{"steps": n}. Use --system-theme light|dark|none to supply the initial OS appearance, and
script events such as {"system_theme": "dark"} to change it on subsequent frames.
Appearance defaults to Dark. ui.theme {"mode":"system"} follows the appearance supplied by
the native or web integration at launch and on later frames, while the saved choice stays System.
When the integration cannot report an appearance, as on Linux, System reads the desktop's
settings portal (org.freedesktop.appearance color-scheme) and follows its changes; without
a portal answer it uses Dark. Manual Light and Dark stay fixed when the OS appearance changes.
An offscreen render without --system-theme supplies no OS appearance and therefore resolves
System to Dark, or on Linux to the desktop's portal setting when one is available.
view.scroll_into_view {"track":"Bass"} (a track id or name; without it, the first selected
track) brings the track to the top of the Edit window and the left of the Mix window, the same as
Scroll Into View in the track name menu. A window not on screen catches up when it is next shown.
Panel docking
Tracks and Groups, Clips, MIDI Editor, Memory Locations, and Undo History use movable panels. Commands target the active Edit or Mix view; floating panels stay inside the current viewport. Edit and Mix retain separate arrangements; Clips and MIDI Editor belong to Edit. Transport, plug-in, and specialist utility windows retain their existing hosts. The document/session view is protected.
The Edit window starts with Tracks and Groups in a 168-point column in a dock at the left edge, Clips in a 230-point column in a dock at the right edge, and the MIDI Editor under the session view between them. Both docks run the window's full height. Mix starts with Tracks and Groups docked at its left edge. Memory Locations and Undo History float. A dock column stacks tab groups top to bottom: a panel dropped on a group's middle joins it as a tab, on its upper or lower third stacks a new group above or below it, and at a column's edge makes a new column. A panel dropped at an empty window edge makes a dock there. Layouts, active tabs, panel visibility, and floating rectangles are saved with the app's existing UI preferences. These operations do not edit document contents.
Panel IDs: tracks, clips, midi, memory, history. The protected center is canvas.
ui.dock accepts operation and panel. Operations are open, close, activate, float (with finite positive rect: [x, y, width, height]), or move (with target and zone: tab|center|left|right|top|bottom|above|below; PascalCase names such as Left also parse). open and activate show a hidden panel at its saved location and bring a visible panel to the front. A panel with no saved location, or whose saved anchor cannot be restored, opens at its default place: Tracks and Groups and Clips join their dock's first group, or make that dock again at its default width when it is gone; the MIDI Editor opens under the session view; Memory Locations and Undo History float at their default rectangles. Center and tab group with the target; side zones split beside it; above and below stack the panel in a new group above or below a floating target's group. ui.dock.reset restores the default arrangement. Invalid moves return an error without changing the layout.
Invoke these UI commands through engine.execute, for example {"command":"ui.dock","params":{"operation":"float","panel":"tracks","rect":[100,100,320,440]}}.
Advanced ui.dock operations use the same validated transaction as pointer gestures:
moveFloating updates rect, resizeSplit accepts a boolean child path and
size ({"Ratio":0.5}, {"FixedFirst":280}, or {"FixedSecond":280}),
setStackOpen takes open, and resizeStack takes a point height or null.
A center move can include before (panel ID or null to append) for tab order.
The equivalent {"action": ...} form accepts a serialized shared docking action;
this is also the form emitted by the renderer. Protected-canvas and app-specific panel
rules apply equally to both forms.
The saved UI preferences and window configurations carry docking_version. Settings with no
version, or an older one, hold the split-frame layouts of earlier releases: they load with the
default Edit and Mix layouts and keep every other setting.