songtrace

Turn a song and its lyrics into a playable UltraStar chart.

A CLI and minimal web UI that turns a song — title, artist, lyrics you supply, and a YouTube URL or local video file — into a playable UltraStar-format song folder. Also compatible with Melody Mania.

The pipeline:

1. acquire — download audio/video via yt-dlp (or use a local video file)
2. separate — split vocals/instrumental with Demucs
3. transcribe — send the vocal stem to your own intranet Whisper server (OpenAI-compatible, or onerahmet/openai-whisper-asr-webservice), as a sanity check that the supplied lyrics match the audio and to locate vocal sections
4. align — forced-align the supplied reference lyrics against the vocal stem with wav2vec2 (torchaudio's MMS_FA) to get word timings
5. pitch — track pitch on the vocal stem with torchcrepe
6. notes — turn word timings + pitch into UltraStar notes
7. grid — quantise notes to a beat grid
8. package — write the final song folder (.txt + .mp3 + video + cover)

The shipped .mp3 is the full original mix (vocals included) — the separated stems are used for analysis only and are not included in the output.

The output is a best-effort draft, not a finished song. Every run ends by printing the low-confidence regions it found (e.g. 3 low-confidence notes near 0:42–0:47) — open the result in an UltraStar editor such as Yass or USDX and touch those spots up before playing.

Reviewing a song

The web UI plays the finished song folder back against the review, so you can check the timing without loading it into a game first. Pick a song in the queue and the review panel gets the packaged video with a transport above the lyrics; as it plays, the line being sung is highlighted, words light up as they are reached, and a playhead sweeps each line against its notes. Click any line or word to play from there.

Playback is modelled on the game: the #MP3 is the audio and the clock, and the #VIDEO is muted background slaved to it. It also uses the same seconds the notes were generated from, so a word that lights up late in the browser is late in the game too.

The packaged video has its audio track stripped for the same reason. It is the same recording as the #MP3, so a player that does not mute the video plays both at once, a few frames apart, which is heard as an echo. Songs built before this change still have it — check with ffprobe <song>.mp4 for an audio stream, and rebuild with Restart from set to package to clear it without re-downloading.

Editing a song

Above the lyrics, the note sheet shows one line at a time on a pitch-by-beat grid, with the pitch songtrace detected drawn over the notes: where a note is right the curve runs through it, so a wrong one stands out. Fix it there and SAVE LINE; the song folder's chart is rewritten on the spot.

Drag a note to move it and its ends to resize it; everything snaps to the chart's beat grid and to semitones. Double-click empty space to add a note, or a note to change its syllable. With the sheet focused: ←/→ move, ↑/↓ change pitch (Shift for an octave), Shift+←/→ change length, Tab selects the next note, S splits, M merges with the next note, G/F/R make a note golden, freestyle or rap, Delete removes, Ctrl+Z undoes, Ctrl+S saves and P plays the line.

The ↵ markers either side of a line are its line breaks: drag one to move it (it snaps to beats and stays between the lines it separates) and double-click one to put it back where songtrace places it. Moving a break does not mark either line's notes as edited, so they still follow a rebuild.

The GRID bar above the notes sets the chart's BPM and GAP. Changes redraw the grid as you type and are charted on APPLY; ×2 and ÷2 fix a tempo detected at double or half speed, and ALIGN TO NOTE moves GAP the least that puts the selected note on a beat. Notes keep their timing in seconds and re-snap to the new grid. RESET returns to the BPM songtrace chose (your BPM override, or the detected tempo) with GAP at the first note.

Edits survive every rebuild, REBUILD ALL FROM… included: they are kept apart from what the pipeline generates and laid back over it each time. REVERT returns a line to the generated notes. Changing a line's lyrics sets its edit aside rather than applying it to different words; the sheet lists those so you can discard them.

Configuration

Config is TOML. Resolution order:

1. --config <path> passed explicitly on the command line
2. the SONGTRACE_CONFIG environment variable
3. ~/.config/songtrace/config.toml

Copy the example and fill in your Whisper server:

cp config.example.toml config.toml
$EDITOR config.toml   # set whisper_url at minimum

See config.example.toml for every available field (whisper_url, whisper_api_key, whisper_model, whisper_api, output_dir, work_dir, creator, demucs_model, crepe_model, device, bind_host, port, ytdlp_clients, ytdlp_cookies, ytdlp_impersonate, ytdlp_pot_provider_url, ytdlp_version).

whisper_api selects the wire format: "openai" (default) speaks the OpenAI-compatible POST {whisper_url}/audio/transcriptions API; "asr-webservice" speaks onerahmet/openai-whisper-asr-webservice's POST {whisper_url}/asr API instead (point whisper_url at that server's base URL with no /v1 suffix — whisper_model/whisper_api_key are unused for this backend, since the model is fixed by how that server was started).

Docker

cp config.example.toml config.toml
$EDITOR config.toml
docker compose up -d

The web UI is then at http://localhost:8000. Downloaded model weights (Demucs, wav2vec2, ...) are cached under ./data on the host so they survive container recreation.

Run the CLI inside the container instead of the web UI:

docker compose run --rm songtrace songtrace make \
  --title "Song Title" --artist "Artist Name" \
  --lyrics /data/lyrics.txt --youtube https://youtu.be/...

GPU passthrough requires the NVIDIA Container Toolkit installed on the host; docker-compose.yml requests one GPU via the nvidia device driver.

Sharing the GPU

songtrace serve is long-lived but only wants the GPU in bursts, so it hands it back after every stage rather than holding it for the life of the process. Three stages load a model — separate (Demucs), align (wav2vec2) and pitch (torchcrepe) — and each one's weights are dropped as soon as that stage finishes, followed by torch.cuda.empty_cache() to return the allocator's freed blocks to the driver instead of keeping them for reuse. This matters most between separate and align: transcribe sits between them waiting on your Whisper server, which can be minutes of a GPU nothing is using.

Stages that gave memory back say so in the job log:

[separate] released 3704 MiB of GPU memory

The cost is reloading each model once per job instead of once per process, which is seconds against stages that run for minutes. What this does not release is the CUDA context itself, a few hundred MB that belongs to the process until it exits; freeing that would mean running each stage in a subprocess. If you need every last MB back, stop the container between runs.

Bare-metal

pip install -e .
songtrace make --help

songtrace make --title TITLE --artist ARTIST --lyrics LYRICS_FILE \
  (--youtube URL | --video PATH) \
  [--genre GENRE] [--year YEAR] [--language LANGUAGE] [--cover COVER_IMAGE] \
  [--bpm BPM] [--force-from STAGE] [--config CONFIG]

Run the web UI instead with songtrace serve [--config CONFIG].

Re-running a stage

Each stage caches its output under work_dir. If a later stage needs another pass (e.g. you want to retry alignment with different lyrics, or pitch tracking looks off), re-run with --force-from <stage> to redo that stage and everything after it, reusing the cached output of everything before it:

songtrace make --title "..." --artist "..." --lyrics lyrics.txt \
  --youtube https://youtu.be/... --force-from align

Valid stages, in pipeline order: acquire, separate, transcribe, align, pitch, notes, grid, package.

In the web UI this is what selecting a queue entry does: its title, artist, lyrics and the rest come back up in the form, so you can adjust them and hit REBUILD SONG. Only the stages your edit actually invalidates are rerun — fixing a line of lyrics realigns without re-downloading or re-separating. CREATE SONG, at the top of the queue, clears the page to start a fresh one.

That planning works off the fields you changed, which cannot see a change in songtrace itself: after an upgrade, a rebuild of an older song finds every stage cached and does nothing. Restart from on the rebuild form is the way to redo a stage anyway — pick package to repackage without re-downloading. It can only widen what is rerun, so an edit that invalidates an earlier stage still wins.

REBUILD ALL FROM…, under CREATE SONG, does the same for every song at once: pick the stage an update improved and each song is rebuilt from there with the inputs it has now. Every song starts ticked; untick any already rebuilt by hand (each shows when it was last built). Songs still in progress are skipped, as are songs whose uploaded video is gone when the stage is acquire.

YouTube download failures

HTTP Error 403: Forbidden from the acquire stage means YouTube served the page but refused the media URL. In rough order of likelihood:

1. yt-dlp is stale. YouTube changes its defences every few weeks, so songtrace keeps yt-dlp current itself. Once a day, and again whenever a download fails outright, it checks PyPI for a newer stable release, installs it under $XDG_CACHE_HOME/songtrace/ (/data/models/cache in the container), and tries it before the copy installed with songtrace. That copy is never modified and is always the fallback, so a release that breaks compatibility costs a retry rather than the download. The job log names the yt-dlp version behind each attempt.

   ytdlp_version overrides this: "bundled" uses only the installed copy, and a release number such as "2026.08.19" pins that release instead of the newest.

2. Missing JS runtime. yt-dlp needs yt-dlp-ejs plus a JavaScript engine to solve YouTube's player challenges; without them it silently falls back to formats that 403. The [default,curl-cffi,deno] extras in pyproject.toml cover this, for both the installed copy and the ones fetched at runtime. Check with docker compose exec songtrace yt-dlp -v --simulate <url> — the debug header must show a JS runtimes: line that is not none.

3. The IP is rate-limited or flagged. Datacenter and VPN ranges get throttled hardest. Try from another network before changing config.

4. A PO token is required. If the log says so explicitly, run bgutil-ytdlp-pot-provider as a sidecar, install its plugin into the image, and point ytdlp_pot_provider_url at it.

ytdlp_clients retries the download against each listed player client in turn, so a client that YouTube has broken is skipped automatically. ytdlp_cookies (a Netscape cookie jar, for age-gated videos) and ytdlp_impersonate (a browser TLS fingerprint, e.g. "chrome") are the other two levers. Do not use a cookie jar from an account you care about — YouTube bans for automated access.

Failed downloads leave the work dir intact, so re-running the job resumes rather than starting over.

Regenerating the web UI screenshot

scripts/screenshot_webui.py seeds the real app with a fixed set of jobs covering every status and screenshots it to docs/images/webui.png. It uses ffmpeg to give the selected song a playable folder so the shot includes the player; without ffmpeg on PATH it still runs, just without it.

pip install -e '.[dev]'
playwright install chromium
python scripts/screenshot_webui.py

License

AGPLv3