branderson
f187bfd6c0
songtrace serve is long-lived but only wants the GPU in bursts, and until now it kept everything it ever loaded. align cached wav2vec2 in a module dict that was never cleared, torchcrepe caches its model on its own infer() function, and even once the last reference to a model goes, PyTorch keeps the freed blocks in its allocator rather than returning them to the driver. On a shared box that is the whole GPU held against one idle process. Stages now hand it back as soon as they are done with it. A stage that caches a model registers how to drop it with @on_release, so models.py needs to know nothing about the pipeline, and run_pipeline calls release() in a finally after each stage — not once per job, because transcribe sits between separate and align waiting on the Whisper server, which can be minutes of a GPU nothing is using. release() answers the bytes it handed back and the stage logs a line when that is not zero, which is the only part of this visible from outside. It will not initialise CUDA just to clean up after a run that never touched it. The cost is reloading each model once per job rather than once per process, seconds against stages that run for minutes. The CUDA context itself stays until the process exits; releasing that would mean a subprocess per stage.
songtrace
Turn a song and its lyrics into a playable UltraStar chart.
A CLI and minimal web UI that turns a song — title, artist, lyrics you supply, and a YouTube URL or local video file — into a playable UltraStar-format song folder. Also compatible with Melody Mania.
The pipeline:
- acquire — download audio/video via
yt-dlp(or use a local video file) - separate — split vocals/instrumental with Demucs
- transcribe — send the vocal stem to your own intranet Whisper server (OpenAI-compatible, or onerahmet/openai-whisper-asr-webservice), as a sanity check that the supplied lyrics match the audio and to locate vocal sections
- align — forced-align the supplied reference lyrics against the vocal stem with wav2vec2 (torchaudio's MMS_FA) to get word timings
- pitch — track pitch on the vocal stem with torchcrepe
- notes — turn word timings + pitch into UltraStar notes
- grid — quantise notes to a beat grid
- package — write the final song folder (
.txt+.mp3+ video + cover)
The shipped .mp3 is the full original mix (vocals included) — the
separated stems are used for analysis only and are not included in the
output.
The output is a best-effort draft, not a finished song. Every run ends by
printing the low-confidence regions it found (e.g.
3 low-confidence notes near 0:42–0:47) — open the result in an UltraStar
editor such as Yass or
USDX and touch those spots up before playing.
Configuration
Config is TOML. Resolution order:
--config <path>passed explicitly on the command line- the
SONGTRACE_CONFIGenvironment variable ~/.config/songtrace/config.toml
Copy the example and fill in your Whisper server:
cp config.example.toml config.toml
$EDITOR config.toml # set whisper_url at minimum
See config.example.toml for every available field (whisper_url,
whisper_api_key, whisper_model, whisper_api, output_dir, work_dir,
creator, demucs_model, crepe_model, device, bind_host, port,
ytdlp_clients, ytdlp_cookies, ytdlp_impersonate,
ytdlp_pot_provider_url).
whisper_api selects the wire format: "openai" (default) speaks the
OpenAI-compatible POST {whisper_url}/audio/transcriptions API; "asr-webservice"
speaks onerahmet/openai-whisper-asr-webservice's
POST {whisper_url}/asr API instead (point whisper_url at that server's base
URL with no /v1 suffix — whisper_model/whisper_api_key are unused for
this backend, since the model is fixed by how that server was started).
Docker
cp config.example.toml config.toml
$EDITOR config.toml
docker compose up -d
The web UI is then at http://localhost:8000. Downloaded model weights
(Demucs, wav2vec2, ...) are cached under ./data on the host so they survive
container recreation.
Run the CLI inside the container instead of the web UI:
docker compose run --rm songtrace songtrace make \
--title "Song Title" --artist "Artist Name" \
--lyrics /data/lyrics.txt --youtube https://youtu.be/...
GPU passthrough requires the NVIDIA Container
Toolkit
installed on the host; docker-compose.yml requests one GPU via the
nvidia device driver.
Sharing the GPU
songtrace serve is long-lived but only wants the GPU in bursts, so it hands
it back after every stage rather than holding it for the life of the process.
Three stages load a model — separate (Demucs), align (wav2vec2) and
pitch (torchcrepe) — and each one's weights are dropped as soon as that
stage finishes, followed by torch.cuda.empty_cache() to return the
allocator's freed blocks to the driver instead of keeping them for reuse. This
matters most between separate and align: transcribe sits between them
waiting on your Whisper server, which can be minutes of a GPU nothing is
using.
Stages that gave memory back say so in the job log:
[separate] released 3704 MiB of GPU memory
The cost is reloading each model once per job instead of once per process, which is seconds against stages that run for minutes. What this does not release is the CUDA context itself, a few hundred MB that belongs to the process until it exits; freeing that would mean running each stage in a subprocess. If you need every last MB back, stop the container between runs.
Bare-metal
pip install -e .
songtrace make --help
songtrace make --title TITLE --artist ARTIST --lyrics LYRICS_FILE \
(--youtube URL | --video PATH) \
[--genre GENRE] [--year YEAR] [--language LANGUAGE] [--cover COVER_IMAGE] \
[--bpm BPM] [--force-from STAGE] [--config CONFIG]
Run the web UI instead with songtrace serve [--config CONFIG].
Re-running a stage
Each stage caches its output under work_dir. If a later stage needs
another pass (e.g. you want to retry alignment with different lyrics, or
pitch tracking looks off), re-run with --force-from <stage> to redo that
stage and everything after it, reusing the cached output of everything
before it:
songtrace make --title "..." --artist "..." --lyrics lyrics.txt \
--youtube https://youtu.be/... --force-from align
Valid stages, in pipeline order: acquire, separate, transcribe,
align, pitch, notes, grid, package.
In the web UI this is what selecting a queue entry does: its title, artist, lyrics and the rest come back up in the form, so you can adjust them and hit REBUILD SONG. Only the stages your edit actually invalidates are rerun — fixing a line of lyrics realigns without re-downloading or re-separating. NEW SONG INSTEAD clears the form to start a fresh song.
YouTube download failures
HTTP Error 403: Forbidden from the acquire stage means YouTube served the
page but refused the media URL. In rough order of likelihood:
-
yt-dlp is stale. YouTube changes its defences every few weeks and the version baked into the image does not. Hotfix a running container, then rebuild so it sticks:
docker compose exec songtrace pip install -U 'yt-dlp[default,curl-cffi,deno]' docker compose restart songtrace docker compose build --pull songtrace # permanent -
Missing JS runtime. yt-dlp needs
yt-dlp-ejsplus a JavaScript engine to solve YouTube's player challenges; without them it silently falls back to formats that 403. The[default,curl-cffi,deno]extras inpyproject.tomlcover this. Check withdocker compose exec songtrace yt-dlp -v --simulate <url>— the debug header must show aJS runtimes:line that is notnone. -
The IP is rate-limited or flagged. Datacenter and VPN ranges get throttled hardest. Try from another network before changing config.
-
A PO token is required. If the log says so explicitly, run bgutil-ytdlp-pot-provider as a sidecar, install its plugin into the image, and point
ytdlp_pot_provider_urlat it.
ytdlp_clients retries the download against each listed player client in
turn, so a client that YouTube has broken is skipped automatically.
ytdlp_cookies (a Netscape cookie jar, for age-gated videos) and
ytdlp_impersonate (a browser TLS fingerprint, e.g. "chrome") are the other
two levers. Do not use a cookie jar from an account you care about — YouTube
bans for automated access.
Failed downloads leave the work dir intact, so re-running the job resumes rather than starting over.
Regenerating the web UI screenshot
scripts/screenshot_webui.py seeds the real app with a fixed set of jobs
covering every status and screenshots it to docs/images/webui.png:
pip install -e '.[dev]'
playwright install chromium
python scripts/screenshot_webui.py