align: let a star token absorb non-lyric audio between lines #25

2026-10-10T19:21:38-07:00

dejima-agent commented

2026-10-10 19:21:38 -07:00

Summary generated by LLM agent

Stacked on #24: it targets that branch so the diff here is only this change. Merge #24 first.

Non-lyric sounds between sung lines, such as background speech or laughter, desynced the alignment. In one song, Whisper transcribed 30 words of background speech over 269–298s that match nothing in the lyrics, and later turned laughter into 31 heard words paired against 5 lyric words. The chart showed a burst of tiny notes over the speech, then the next lines rushed through until a later line pulled it back into sync.

Cause

Forced alignment has to account for every frame of a section with the lyrics' characters or blanks. Speech and laughter look like letters to the model, so the next line's characters get placed on them and the real line is squeezed into whatever time is left.

Fix

MMS_FA, the aligner songtrace uses, has a star token that matches any audio. torchaudio appends it with a log-probability of 0, and songtrace already loads the model with it enabled. _align_section now puts a star before, between and after the lines of each section, and leaves the stars' spans out of the result. Non-lyric audio between lines can then go to a star instead of claiming lyric characters. ctc-forced-aligner, the common wrapper around this same model, does this by default (star_frequency="segment").

Limits:

Stars sit only between lines, so speech that overlaps a sung line still competes with it.
This changes alignment for every song, not just ones with speech. Lines are freer to settle where their letters match best instead of being stretched to fill the audio between them. That should help, but it hasn't been measured.

Needs a GPU A/B before merging

I didn't run the model for this. Please rebuild a few songs with Restart from set to align on this branch and on #24:

the song with the background speech and laughter: the tiny-note burst over the speech should be gone, and the lines after it should no longer be rushed
one or two songs that already align well, to check nothing regresses

Testing

New test: the token sequence for a section is * line * line *, with no star added for a line that has no alignable words, and no word is given a star's span.
Existing align tests pass unchanged.
Full suite passes, except test_version (needs the package installed).

**Summary generated by LLM agent** Stacked on #24: it targets that branch so the diff here is only this change. Merge #24 first. Non-lyric sounds between sung lines, such as background speech or laughter, desynced the alignment. In one song, Whisper transcribed 30 words of background speech over 269–298s that match nothing in the lyrics, and later turned laughter into 31 heard words paired against 5 lyric words. The chart showed a burst of tiny notes over the speech, then the next lines rushed through until a later line pulled it back into sync. ## Cause Forced alignment has to account for every frame of a section with the lyrics' characters or blanks. Speech and laughter look like letters to the model, so the next line's characters get placed on them and the real line is squeezed into whatever time is left. ## Fix MMS_FA, the aligner songtrace uses, has a star token that matches any audio. torchaudio appends it with a log-probability of 0, and songtrace already loads the model with it enabled. `_align_section` now puts a star before, between and after the lines of each section, and leaves the stars' spans out of the result. Non-lyric audio between lines can then go to a star instead of claiming lyric characters. `ctc-forced-aligner`, the common wrapper around this same model, does this by default (`star_frequency="segment"`). Limits: - Stars sit only between lines, so speech that overlaps a sung line still competes with it. - This changes alignment for every song, not just ones with speech. Lines are freer to settle where their letters match best instead of being stretched to fill the audio between them. That should help, but it hasn't been measured. ## Needs a GPU A/B before merging I didn't run the model for this. Please rebuild a few songs with **Restart from** set to `align` on this branch and on #24: - the song with the background speech and laughter: the tiny-note burst over the speech should be gone, and the lines after it should no longer be rushed - one or two songs that already align well, to check nothing regresses ## Testing - New test: the token sequence for a section is `* line * line *`, with no star added for a line that has no alignable words, and no word is given a star's span. - Existing align tests pass unchanged. - Full suite passes, except `test_version` (needs the package installed).

dejima-agent added 1 commit 2026-10-10 19:21:39 -07:00

align: let a star token absorb non-lyric audio between lines 7f7bd6e24a

The aligner had to explain every frame of a section with the lyrics, so
background speech or laughter between lines pulled the next line's
letters onto it, which then came out as a burst of tiny notes before the
real line was rushed to catch up. MMS_FA has a star token that matches
any audio; put one before, between and after the lines of each section,
as ctc-forced-aligner does by default, and drop the stars' spans from the
result.

branderson approved these changes 2026-10-10 20:56:11 -07:00

branderson merged commit a222c352c7 into unheard-lines

2026-10-10 20:56:17 -07:00

branderson referenced this issue from a commit

2026-10-15 14:17:08 -07:00

Merge pull request 'align: let a star token absorb non-lyric audio between lines' (#25) from line-stars into unheard-lines

dejima-agent referenced this issue from a commit

2026-10-15 17:44:37 -07:00

Merge pull request 'align: let a star token absorb non-lyric audio between lines' (#25) from line-stars into unheard-lines

Sign in to join this conversation.