Publishing a prebuilt image can wait. Remove the GitHub Actions image
workflow and the IMAGE_PREFIX/IMAGE_TAG image name in docker-compose.yml,
and have the README's quick start build with docker compose up -d
--build, which also rebuilds after a git pull. The mirror's docs and
script no longer mention Actions or the package.
mirror/github-mirror.sh publishes main and the tags on it to a read-only
GitHub mirror, as one extra commit that adds a notice under the README
title, so every forge commit is unchanged there and each sync is a
fast-forward. A systemd user service and hourly timer run it from a host
that can push to GitHub; --settings turns off issues, pull requests and
the rest there.
A GitHub Actions workflow builds the image on the mirror and pushes it to
ghcr.io, freeing runner disk first since the CUDA build of torch is
several GB. docker-compose.yml names the image after IMAGE_PREFIX and
IMAGE_TAG, so setting them pulls the published image instead of
building, and the README says how.
The README described a CLI with a minimal web UI and told people to fix
charts in Yass. Rewrite it around how songtrace is used now: what it
does, its requirements up front (an NVIDIA GPU, tested on 12 GB, Docker
with the NVIDIA Container Toolkit, a Whisper server of your own, disk for
models and per-song files), a Docker quick start including writing songs
straight into a game's collection, then reviewing, editing and
rebuilding in the web UI. The command line moves near the end.
Regenerate the screenshot to show the note sheet: the screenshot script
now seeds the featured song with syllables, a chart, a pitch track and a
line break, and always scrolls to the review. The example config no
longer sets a real person as #CREATOR, and uses a reserved example host
for Whisper.
Since breaks could be moved, the sheet widened its window to show the
breaks either side of a line. Before an instrumental passage the break
sits mid-gap, sometimes 15s away, so the line's notes were crushed into
a corner of the sheet. The window now grows at most 3s past the line to
show a break; one further off is pinned to the edge, labelled with an
arrow toward where it lies, and still drags inward.
Line breaks were always placed by songtrace, midway between lines and
recomputed around edited ones, so a break could not be put where a singer
breathes. The sheet now draws the breaks either side of a line as markers
that drag along the beat grid, kept between the lines they separate, and
double-click back to automatic. Break moves share the line's undo history
and SAVE LINE, which saves the notes only if they changed, so moving a
break never locks either line's notes.
Moved breaks are stored in edits.json keyed by the line each comes before,
follow their lines when the lyrics change, and win over the computed break
on every rebuild, clamped so a break never lands inside a line's notes.
The review lists each break with that line and whether it was moved, and
PUT /api/jobs/{id}/breaks takes {line: seconds | null}. merge now passes
untouched notes through as they are rather than rebuilding them, which
added float noise to their durations.
The beat grid came from the detected tempo (or the BPM override) and a GAP
pinned to the first note, so it did not always line up with the music,
and snapping did not land on its beats. A GRID bar on the sheet now sets
both, redrawing the grid live as values are tried, with x2 and /2 for a
tempo detected at double or half speed and ALIGN TO NOTE to move GAP the
least that puts the selected note on a beat. Note editing pauses while a
trial grid is unapplied, so notes never snap to a grid that is then
cancelled.
The grid is stored with the song's edits, so every rebuild keeps it, and
grid charts with it in place of its own choice; it records that choice in
tempo.json for RESET to return to. A GAP after the first note moves back
by whole beats, as notes on negative beats are rejected by players. PUT
and DELETE /api/jobs/{id}/grid set and reset it, rewriting the charts in
place.
The sheet's controls were listed in one run-on line under it, which was
hard to scan and grew with every feature. Replace it with a panel grouped
the way an editor's reference card is (select, move, length, notes, kind,
syllable, line) with keys drawn as key caps, opened by a KEYS button or ?,
closed by Esc, and remembered open or closed across visits.
The list is data (SHORTCUTS), and a test checks every letter key it names
is one the sheet's key handler handles, so the two cannot drift apart.
A song's edits are kept through any rebuild, but nothing said which songs
had them. GET /api/jobs and GET /api/rebuild-all now report each song's
edited and set-aside lines, read from its edits file, on the job standing
for the song only, since the runs it replaced would claim the same edits.
Queue cards show them as a badge, and the rebuild dialog lists them per
song and says hand edits are kept.
Notes on the sheet can now be selected and edited: dragged in time and
pitch, resized from either end, nudged and stretched from the keyboard,
split, merged, added, deleted, given a syllable or a kind (golden,
freestyle, rap), with undo/redo. Everything snaps to the chart's beat grid
and to semitones, as the chart itself will. SAVE LINE stores the line
through the edits API, which rewrites the charts in place; REVERT goes back
to the generated notes; a line's edits set aside because its lyrics changed
are listed with a way to discard them. Unsaved changes are not lost to
paging, following playback, switching songs or closing the tab.
The operations live in static/edit-model.js as pure functions, unit-tested
with node --test. The sheet hit-tests by geometry rather than event
targets, since a render replaces the elements under the pointer.
The review drew each line's notes as thin bars in a 20px strip, with no
pitch axis or beat grid, so it was hard to tell where a note sat. Add a
sheet that shows one line at a time the way an UltraStar editor does: a
row per semitone, shaded like piano keys and named in a gutter, the
chart's own beat grid with bars every 4 and 16 beats, notes as blocks
styled by kind (golden, freestyle, rap) with low-confidence ones outlined,
neighbouring lines faded in for context, syllables and aligned words
beneath, and a playhead that follows the transport, paging with it.
The detected pitch curve is drawn over the notes, served by a new GET
/api/jobs/{id}/pitch in note semitones, so a note off the sung pitch
stands out. The sheet is its own ES module; its geometry is unit-tested
with node --test, run from pytest and skipped where Node is absent.
The groundwork for an in-app editor. Edits live in edits.json in the work
dir, which no stage produces, so no rebuild removes them. An edit is a
whole lyric line as the user left it, in seconds, so a rebuild with a new
BPM re-snaps it. grid lays the edits over the freshly generated notes
(neighbouring lines give way to an edited one), writes the result to
notes_final.json with warnings brought up to date, and charts that. When
the lyrics change, edits follow their lines by text and order; an edit
whose line's words changed is set aside as stale and reported, not applied.
Notes gain a kind, so golden, freestyle and rap notes can be charted. The
review endpoint returns the merged notes with their syllables, kinds and
edit state, plus the chart's beat grid. PUT/DELETE
/api/jobs/{id}/lines/{n} save and revert a line, DELETE
/api/jobs/{id}/stale/{n} discards a stale edit; each rewrites the work and
packaged charts in place with the tempo already chosen, so a save
re-encodes nothing.
A fixed 0.5 alignment-score cutoff flagged 73-99% of notes on sung
vocals, so the flag pointed nowhere. Scores still separate bad words from
good ones, but their scale differs from song to song: flag a word below a
quarter of the song's median score, or one shorter than 0.08s or longer
than 3s, which no singer produces and which a word stretched over
something else does. Flagged notes less than 2s apart share one warning.
The worker set status "done" before removing the job's upload and before
setting finished_at, so a client polling for done could see the upload
still there and finished_at null. Record the outcome and publish status,
error and finished_at together, under the lock, as the last step.
Debian bookworm's python3-venv seeds pip 23.0.1, which rejects the
PyTorch CPU index's typing_extensions wheel because its metadata spells
the name with an underscore. It then falls back to the sdist, whose build
needs flit_core, which that index does not carry.
The aligner had to explain every frame of a section with the lyrics, so
background speech or laughter between lines pulled the next line's
letters onto it, which then came out as a burst of tiny notes before the
real line was rushed to catch up. MMS_FA has a star token that matches
any audio; put one before, between and after the lines of each section,
as ctc-forced-aligner does by default, and drop the stars' spans from the
result.
A line with nothing in the transcript was always dropped as not sung. When
Whisper skips a passage (one song lost a whole verse, 50s of clear singing
with no segments), that drops lines that were sung. Find singing in the
vocal stem that no segment covers; a run of unmatched lines with such
singing between its heard neighbours is kept, and that singing joins the
alignment sections so those lines have somewhere to go. Only singing that
kept a line is added, so an untranscribed ad-lib draws no lines into it.
The dialog now lists the songs it would rebuild, all ticked, each with when
it was last built and whether that was a rebuild, so songs already rebuilt
by hand can be unticked. GET /api/rebuild-all returns that list, so the
page no longer repeats the server's rule for which job stands for a song,
and POST takes the ticked job ids. A selected job that has since been
rebuilt is reported as skipped rather than rebuilt twice.
When an update improves a stage, existing songs only pick it up when each
is rebuilt from that stage. POST /api/rebuild-all queues an ordinary
rebuild with unchanged inputs for the current job of every song: jobs that
were themselves rebuilt are skipped, as are songs still in progress and
songs whose upload is gone when the stage is acquire. The sidebar button
opens a dialog to pick the stage and confirm the count.
Runs on pull_request_target so the workflow always comes from main, on the
confined runner tier. Installs CPU-only torch before the package so pip
does not pull the CUDA build, then runs pytest. .dejima/allowed-domains
declares the hosts the job fetches from.
Separated stems keep faint bleed during instrumental passages, and crepe
finds a pitch in it with enough confidence to pass the voicing mask. A word
whose alignment ran into such a gap was then filled with notes for the
whole gap. Zero the confidence of frames more than 25 dB below the song's
90th-percentile A-weighted loudness, relative so it holds at any mastering
level.
The align, transcribe, lyrics and notes tests quoted lines from published
songs, and the UltraStar parser fixtures were complete community charts of
commercial songs. Replace them with counting words and two hand-written
charts that keep what the parser tests rely on: CRLF line endings,
Latin-1 bytes, a decimal-comma BPM, freestyle/golden/rap notes, negative
pitch and a two-number line break.
A missing span covers whole Whisper segments, so it often also holds lines
that were heard and matched. Skipping every aligned word starting inside one
dropped those lines' notes and left empty line breaks in the chart. Words
from lines never heard are already excluded via dropped_lines, so the span
now only drives the warning and the review marker.
YouTube breaks old yt-dlp releases faster than songtrace is released.
songtrace now checks PyPI daily, and again after a download fails outright,
installs a newer stable release into its cache dir and tries it before the
bundled copy, which is never modified. yt-dlp runs in a subprocess through
its CLI, all from one module, so a cached release never mixes with the
imported one. ytdlp_version pins a release or turns updating off.
songtrace serve is long-lived but only wants the GPU in bursts, and until now
it kept everything it ever loaded. align cached wav2vec2 in a module dict that
was never cleared, torchcrepe caches its model on its own infer() function,
and even once the last reference to a model goes, PyTorch keeps the freed
blocks in its allocator rather than returning them to the driver. On a shared
box that is the whole GPU held against one idle process.
Stages now hand it back as soon as they are done with it. A stage that caches
a model registers how to drop it with @on_release, so models.py needs to know
nothing about the pipeline, and run_pipeline calls release() in a finally
after each stage — not once per job, because transcribe sits between separate
and align waiting on the Whisper server, which can be minutes of a GPU nothing
is using.
release() answers the bytes it handed back and the stage logs a line when that
is not zero, which is the only part of this visible from outside. It will not
initialise CUDA just to clean up after a run that never touched it.
The cost is reloading each model once per job rather than once per process,
seconds against stages that run for minutes. The CUDA context itself stays
until the process exits; releasing that would mean a subprocess per stage.
Two problems found in review.
CREATE SONG emptied the form but left the previous song's review, player,
output path and queue selection in place, which reads as though the blank
form and that song belong together. It now clears the page down to a blank
sheet, and a `composing` flag stops the next poll selecting the newest song
back into it.
The reason it cleared so little is that clearDetail reached for an element id
that no longer exists, so it threw partway through and left everything after
the first two lines untouched. It now targets the review box and takes the
reason to show, since "no songs queued yet" is wrong when you have simply
deselected.
Separately, rebuild plans from the fields that changed, which cannot see a
change in songtrace itself: rebuilding a song packaged before the video's
audio was stripped found every stage cached and kept the echo. The rebuild
form gains "restart from", the CLI's --force-from, which combines with the
planner rather than replacing it — an explicit stage may widen what is rerun
but never narrow it below what the edit requires.
Also: a browser submits a textarea with CRLF, so every rebuild from the page
looked like a lyrics edit and redid the alignment. Lyrics are normalised to LF
on the way in. And [hidden] is only display:none, which .field's display:flex
was winning, so the new control showed in new-song mode despite the property
being set.
Checking whether the notes line up meant packaging the song, finding it in
the game and starting it there. The review panel now plays the song folder
itself: a transport above the lyrics, the line being sung highlighted, words
lighting up as they are reached, and a playhead sweeping each line against
its notes. Clicking a line or a word plays from there.
Playback is modelled on the game rather than on whatever is convenient: #MP3
is the audio and the clock, #VIDEO is muted background slaved to it, and the
times are the same seconds the notes were generated from. A word that lights
up late here is late in the game.
package now strips the audio track from the video it writes. It is the same
recording as the #MP3, so a player that does not mute the video plays both a
few frames apart, which is heard as an echo. The compatible-video branch
remuxes with -c copy -an rather than copying the file, so it still does not
re-encode. The browser player mutes the video as well, which covers song
folders built before this.
A new /api/jobs/{id}/media/{audio,video,cover} serves what the song's own
#MP3/#VIDEO/#COVER headers name, refusing anything resolving outside the song
folder. Range requests come from FileResponse, so seeking works.
Also, per review: CREATE SONG moves out of the form and into the top of the
queue, where it is always available, and a successful submission keeps the
fields it sent instead of emptying them.
branderson