Alignment runs over the whole song when vocals have no 6s gap #20

New Issue

2026-10-09T18:19:58-07:00

dejima-agent commented

2026-10-09 18:19:58 -07:00

Filed by LLM agent

The align stage force-aligns all lyrics inside each "vocal section". A section only ends at a gap of 6s or more between Whisper segments, and on songs without such gaps the whole song becomes one section (one test song gave a single 0–205s section). A misalignment anywhere can then shift words elsewhere.

Branch anchored-align holds an experiment. It anchors each lyric line to the Whisper segments its words were matched in, and aligns each group of lines only inside its segments' window. On one real song:

median alignment score went from 0.08 to 0.11, and the 90th percentile from 0.31 to 0.54
words under 0.5 went from 99% to 88%
implausibly short words (under 0.1s) increased slightly, from 102 to 110
the bug it was meant to fix turned out to have a different cause (fixed separately in #15)

It isn't worth merging as it stands. Two problems:

Whisper segments rarely end at a line break, so lines that straddle segments chain neighbouring groups together and the windows stay large.
replace blocks are paired proportionally, which is weak. The autojunk problem in the matcher (#16) makes this worse.

A version that anchors individual words to segments rather than whole lines, after #16, might be worth re-measuring. The branch has unit tests. The evaluation re-ran the align stage on CPU with and without the change against the same transcript and vocal stem, and compared word durations, alignment scores and how much of each word's time was voiced.

**Filed by LLM agent** The `align` stage force-aligns all lyrics inside each "vocal section". A section only ends at a gap of 6s or more between Whisper segments, and on songs without such gaps the whole song becomes one section (one test song gave a single 0–205s section). A misalignment anywhere can then shift words elsewhere. Branch `anchored-align` holds an experiment. It anchors each lyric line to the Whisper segments its words were matched in, and aligns each group of lines only inside its segments' window. On one real song: - median alignment score went from 0.08 to 0.11, and the 90th percentile from 0.31 to 0.54 - words under 0.5 went from 99% to 88% - implausibly short words (under 0.1s) increased slightly, from 102 to 110 - the bug it was meant to fix turned out to have a different cause (fixed separately in #15) It isn't worth merging as it stands. Two problems: - Whisper segments rarely end at a line break, so lines that straddle segments chain neighbouring groups together and the windows stay large. - `replace` blocks are paired proportionally, which is weak. The `autojunk` problem in the matcher (#16) makes this worse. A version that anchors individual words to segments rather than whole lines, after #16, might be worth re-measuring. The branch has unit tests. The evaluation re-ran the align stage on CPU with and without the change against the same transcript and vocal stem, and compared word durations, alignment scores and how much of each word's time was voiced.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: branderson/songtrace#20