branderson/songtrace
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

align: keep lines Whisper missed when their singing went untranscribed #24

Merged
branderson merged 4 commits from unheard-lines into main 2026-10-10 22:09:00 -07:00
Conversation 2 Commits 4 Files Changed 4
+239 -26
dejima-agent commented 2026-10-10 18:58:40 -07:00
Collaborator
Copy Link
Copy Source

Summary generated by LLM agent

A song had its whole first verse marked "dropped (not sung)" although every line of it is sung, and rebuilding from scratch gave the same result.

Cause

Whisper returned no segments for about 50s of clear singing at the start of the song. In the vocal stem that stretch sits at -15 to -25 dB, against roughly -87 dB of silence before it. classify_lyrics_match drops any reference line with nothing matched in the transcript, so every line in the verse was dropped. The rule can't tell "the singer skipped this line" from "Whisper skipped this passage".

Fix

The audio can tell them apart. A line that really isn't sung leaves no untranscribed singing between the heard lines on either side of it; a passage Whisper skipped does.

  • align.untranscribed_singing finds stretches of the vocal stem with singing in them but no transcript segment. Singing means frames within 25 dB of the stem's 90th-percentile level, so separation bleed doesn't count. Gaps under 1s are bridged, and stretches under 3s are ignored.
  • classify_lyrics_match takes those stretches as unheard. A run of unmatched lines is only dropped when none of that singing lies between its heard neighbours.
  • recovered_spans returns the singing that kept a line, and it's added to the transcript before vocal_sections, so the kept lines are aligned there. Only singing that kept a line is added: an untranscribed ad-lib with no missing line around it never becomes a section that lines get assigned to.

Checked against the song's transcript and vocal stem, without running any model: the seven verse lines are now kept, and the main section starts at 8.8s instead of 59s. One other dropped line stays dropped, because its neighbours are heard in back-to-back segments with nothing untranscribed between them. A separate 11s stretch of untranscribed singing later in the song has no missing lines around it and is correctly left alone.

Verifying on the GPU machine

Rebuild the song with Restart from set to align. Expect align_meta.json to list one dropped line instead of eight, and the review to show notes for the first verse between roughly 10s and 60s.

Memory

This song's main alignment section grows from 288s to 338s, and the aligner processes a whole section in one pass. A CPU run of that section ran out of memory in my sandbox (about 9 GB). The GPU handled 288s before, but sections well over five minutes may need the emission computed in chunks. Not changed here.

Testing

  • New transcribe tests: lines with untranscribed singing between their neighbours are kept; singing elsewhere keeps nothing; the recovered spans are clipped to the window between heard neighbours.
  • New align tests: the detector finds a 5s sung stretch the transcript skips, ignoring a 1s burst and low-level noise, on generated audio. Lines kept this way are aligned inside that stretch, and nothing is dropped.
  • All test text is synthetic counting words. Nothing from the song is in the repo.
  • Full suite passes, except test_version (needs the package installed).
**Summary generated by LLM agent** A song had its whole first verse marked "dropped (not sung)" although every line of it is sung, and rebuilding from scratch gave the same result. ## Cause Whisper returned no segments for about 50s of clear singing at the start of the song. In the vocal stem that stretch sits at -15 to -25 dB, against roughly -87 dB of silence before it. `classify_lyrics_match` drops any reference line with nothing matched in the transcript, so every line in the verse was dropped. The rule can't tell "the singer skipped this line" from "Whisper skipped this passage". ## Fix The audio can tell them apart. A line that really isn't sung leaves no untranscribed singing between the heard lines on either side of it; a passage Whisper skipped does. - `align.untranscribed_singing` finds stretches of the vocal stem with singing in them but no transcript segment. Singing means frames within 25 dB of the stem's 90th-percentile level, so separation bleed doesn't count. Gaps under 1s are bridged, and stretches under 3s are ignored. - `classify_lyrics_match` takes those stretches as `unheard`. A run of unmatched lines is only dropped when none of that singing lies between its heard neighbours. - `recovered_spans` returns the singing that kept a line, and it's added to the transcript before `vocal_sections`, so the kept lines are aligned there. Only singing that kept a line is added: an untranscribed ad-lib with no missing line around it never becomes a section that lines get assigned to. Checked against the song's transcript and vocal stem, without running any model: the seven verse lines are now kept, and the main section starts at 8.8s instead of 59s. One other dropped line stays dropped, because its neighbours are heard in back-to-back segments with nothing untranscribed between them. A separate 11s stretch of untranscribed singing later in the song has no missing lines around it and is correctly left alone. ## Verifying on the GPU machine Rebuild the song with **Restart from** set to `align`. Expect `align_meta.json` to list one dropped line instead of eight, and the review to show notes for the first verse between roughly 10s and 60s. ## Memory This song's main alignment section grows from 288s to 338s, and the aligner processes a whole section in one pass. A CPU run of that section ran out of memory in my sandbox (about 9 GB). The GPU handled 288s before, but sections well over five minutes may need the emission computed in chunks. Not changed here. ## Testing - New transcribe tests: lines with untranscribed singing between their neighbours are kept; singing elsewhere keeps nothing; the recovered spans are clipped to the window between heard neighbours. - New align tests: the detector finds a 5s sung stretch the transcript skips, ignoring a 1s burst and low-level noise, on generated audio. Lines kept this way are aligned inside that stretch, and nothing is dropped. - All test text is synthetic counting words. Nothing from the song is in the repo. - Full suite passes, except `test_version` (needs the package installed).
dejima-agent added 1 commit 2026-10-10 18:58:41 -07:00
align: keep lines Whisper missed when their singing went untranscribed 62b6c891c8 
A line with nothing in the transcript was always dropped as not sung. When
Whisper skips a passage (one song lost a whole verse, 50s of clear singing
with no segments), that drops lines that were sung. Find singing in the
vocal stem that no segment covers; a run of unmatched lines with such
singing between its heard neighbours is kept, and that singing joins the
alignment sections so those lines have somewhere to go. Only singing that
kept a line is added, so an untranscribed ad-lib draws no lines into it.
branderson added 2 commits 2026-10-10 20:56:19 -07:00
align: let a star token absorb non-lyric audio between lines 7f7bd6e24a 
The aligner had to explain every frame of a section with the lyrics, so
background speech or laughter between lines pulled the next line's
letters onto it, which then came out as a burst of tiny notes before the
real line was rushed to catch up. MMS_FA has a star token that matches
any audio; put one before, between and after the lines of each section,
as ctc-forced-aligner does by default, and drop the stars' spans from the
result.
Merge pull request 'align: let a star token absorb non-lyric audio between lines' (#25) from line-stars into unheard-lines
tests / test (pull_request_target) Failing after 1m32s
Details
a222c352c7 
Reviewed-on: #25
Reviewed-by: branderson <brad@branderson.io>
branderson approved these changes 2026-10-10 21:00:14 -07:00
Dismissed
dejima-agent added 1 commit 2026-10-10 22:04:17 -07:00
Merge remote-tracking branch 'origin/main' into unheard-lines
tests / test (pull_request_target) Successful in 4m9s
Details
be35fe0ddd
branderson approved these changes 2026-10-10 22:08:55 -07:00
branderson merged commit ad15408055 into main 2026-10-10 22:09:00 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
branderson
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
dejima-agent branderson
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: branderson/songtrace#24