Merge pull request 'Mirror main to GitHub' (#37) from github-mirror into main

Reviewed-on: #37
Reviewed-by: branderson <brad@branderson.io>

README.md

@@ -1,175 +1,195 @@
 # songtrace
 
-Turn a song and its lyrics into a playable UltraStar chart.
+Self-hosted karaoke chart builder for [UltraStar](https://usdx.eu/format/) and
-
+compatible games such as [Melody Mania](https://melodymania.org/). Give it a
-A CLI and minimal web UI that turns a song — title, artist, lyrics you supply,
+song's video and lyrics and it charts the notes, timing and pitch, ready for
-and a YouTube URL or local video file — into a playable
+your collection. Fix anything in the built-in editor; your edits survive when
-[UltraStar](https://usdx.eu/format/)-format song folder. Also compatible with
+you rebuild your whole collection after a songtrace update.
-[Melody Mania](https://melodymania.org/).
 
 ![Web UI](docs/images/webui.png)
 
-The pipeline:
+## What it does
 
-1. **acquire** — download audio/video via `yt-dlp` (or use a local video file)
+You give it a title, artist, the lyrics, and a YouTube URL or a video file.
-2. **separate** — split vocals/instrumental with Demucs
+songtrace writes a song folder (`.txt` chart, `.mp3`, video and cover) into
-3. **transcribe** — send the vocal stem to your own intranet Whisper server
+your collection:
-   (OpenAI-compatible, or [onerahmet/openai-whisper-asr-webservice](https://github.com/ahmetoner/whisper-asr-webservice)),
-   as a sanity check that the supplied lyrics match the audio and to locate
-   vocal sections
-4. **align** — forced-align the *supplied* reference lyrics against the vocal
-   stem with wav2vec2 (torchaudio's MMS_FA) to get word timings
-5. **pitch** — track pitch on the vocal stem with torchcrepe
-6. **notes** — turn word timings + pitch into UltraStar notes
-7. **grid** — quantise notes to a beat grid
-8. **package** — write the final song folder (`.txt` + `.mp3` + video + cover)
 
-The shipped `.mp3` is the full original mix (vocals included) — the
+1. **acquire**: downloads the video with `yt-dlp`, or takes your file
-separated stems are used for analysis only and are not included in the
+2. **separate**: splits vocals from the backing track (Demucs)
-output.
+3. **transcribe**: has your Whisper server transcribe the vocals, to check
+   the lyrics match the recording and to find where the singing is
+4. **align**: places each word of *your* lyrics on the vocals (wav2vec2)
+5. **pitch**: tracks the sung pitch (torchcrepe)
+6. **notes**: turns word timings and pitch into notes
+7. **grid**: snaps the notes to a beat grid
+8. **package**: writes the song folder
 
-**The output is a best-effort draft**, not a finished song. Every run ends by
+The result is a strong draft, not a finished chart. The web UI flags the notes
-printing the low-confidence regions it found (e.g.
+it is least sure of, plays the song back against the chart, and lets you fix
-`3 low-confidence notes near 0:42–0:47`) — open the result in an UltraStar
+what it got wrong before you sing it.
-editor such as [Yass](https://www.yass-along.com/) or
-[USDX](https://usdx.eu/) and touch those spots up before playing.
 
-## Reviewing a song
+## Requirements
 
-The web UI plays the finished song folder back against the review, so you can
+- **An NVIDIA GPU with CUDA.** songtrace is developed on a 12 GB card.
-check the timing without loading it into a game first. Pick a song in the
+  Separating the vocals (Demucs) alone takes about 3.7 GB, and alignment holds
-queue and the review panel gets the packaged video with a transport above the
+  a whole stretch of singing in memory at once, so a long song whose vocals
-lyrics; as it plays, the line being sung is highlighted, words light up as
+  never pause needs the most. Cards under 12 GB are untested. Each stage gives
-they are reached, and a playhead sweeps each line against its notes. Click any
+  the GPU back when it finishes (see [Sharing the GPU](#sharing-the-gpu)).
-line or word to play from there.
+  `device = "cpu"` in the config runs without a GPU, but it is untested, far
+  slower, and needs plenty of system RAM.
+- **Docker** with the
+  [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/),
+  so the container can see the GPU.
+- **A Whisper server of your own.** songtrace does not include one. Either an
+  OpenAI-compatible API or
+  [whisper-asr-webservice](https://github.com/ahmetoner/whisper-asr-webservice)
+  works. If it shares songtrace's GPU, it needs its own share of the memory.
+- **Disk space:** a few GB of model weights, downloaded on first use, plus a
+  working folder per song holding its source video and separated tracks,
+  typically several hundred MB.
 
-Playback is modelled on the game: the `#MP3` is the audio and the clock, and
+## Quick start (Docker)
-the `#VIDEO` is muted background slaved to it. It also uses the same seconds
-the notes were generated from, so a word that lights up late in the browser is
-late in the game too.
 
-The packaged video has its audio track stripped for the same reason. It is the
+```sh
-same recording as the `#MP3`, so a player that does not mute the video plays
+cp config.example.toml config.toml
-both at once, a few frames apart, which is heard as an echo. Songs built
+$EDITOR config.toml   # set whisper_url to your Whisper server
-before this change still have it — check with `ffprobe <song>.mp4` for an
+docker compose up -d --build
-audio stream, and rebuild with **Restart from** set to `package` to clear it
+```
-without re-downloading.
 
-## Editing a song
+Open `http://localhost:8000`, fill in the form and press **CREATE SONG**. To
+update, `git pull` and run the same `docker compose` command again.
 
-Above the lyrics, the note sheet shows one line at a time on a pitch-by-beat
+Finished songs go to `output_dir` (`/data/output` in the container). To write
-grid, with the pitch songtrace detected drawn over the notes: where a note is
+them straight into your game's song collection, mount that folder there in
-right the curve runs through it, so a wrong one stands out. Fix it there and
+`docker-compose.yml`:
-**SAVE LINE**; the song folder's chart is rewritten on the spot.
 
-Drag a note to move it and its ends to resize it; everything snaps to the
+```yaml
-chart's beat grid and to semitones. Double-click empty space to add a note,
+    volumes:
-or a note to change its syllable. With the sheet focused: ←/→ move, ↑/↓
+      - ./config.toml:/config/config.toml:ro
-change pitch (Shift for an octave), Shift+←/→ change length, Tab selects the
+      - ./data:/data
-next note, S splits, M merges with the next note, G/F/R make a note golden,
+      - /path/to/your/UltraStar/songs:/data/output
-freestyle or rap, Delete removes, Ctrl+Z undoes, Ctrl+S saves and P plays
+```
-the line.
 
-The ↵ markers either side of a line are its line breaks: drag one to move it
+Everything else lives under `./data` on the host (model weights, and each
-(it snaps to beats and stays between the lines it separates) and double-click
+song's working files), so it survives recreating the container.
-one to put it back where songtrace places it. Moving a break does not mark
-either line's notes as edited, so they still follow a rebuild.
 
-The **GRID** bar above the notes sets the chart's BPM and GAP. Changes redraw
+## Reviewing and editing a song
-the grid as you type and are charted on **APPLY**; ×2 and ÷2 fix a tempo
-detected at double or half speed, and **ALIGN TO NOTE** moves GAP the least
-that puts the selected note on a beat. Notes keep their timing in seconds and
-re-snap to the new grid. **RESET** returns to the BPM songtrace chose (your
-BPM override, or the detected tempo) with GAP at the first note.
 
-Edits survive every rebuild, REBUILD ALL FROM… included: they are kept apart
+Pick a song in the queue. The review panel plays the packaged song with its
-from what the pipeline generates and laid back over it each time. **REVERT**
+video, highlights the line and words being sung, and shows the chart on a note
-returns a line to the generated notes. Changing a line's lyrics sets its edit
+sheet above the lyrics. Playback works like the game: the `#MP3` is the clock
-aside rather than applying it to different words; the sheet lists those so
+and the video follows it muted, so a word that lights up late here is late in
-you can discard them.
+the game too.
+
+The note sheet shows one line at a time on a pitch-by-beat grid, with the
+pitch songtrace detected drawn over the notes. Where a note is right the curve
+runs through it, so a wrong one stands out. Fix it there and **SAVE LINE**; the
+song's chart is rewritten on the spot.
+
+- **Notes:** drag a note to move it, drag its ends to resize it. Everything
+  snaps to the beat grid and to semitones. Double-click empty space to add a
+  note, or a note to change its syllable. With the sheet focused, the arrow
+  keys move notes, S splits, M merges, G/F/R make a note golden, freestyle or
+  rap, Ctrl+Z undoes and Ctrl+S saves. Press **?** for every shortcut.
+- **Line breaks:** the ↵ markers either side of a line. Drag one to move it,
+  and double-click it to hand it back to songtrace.
+- **Tempo:** the **GRID** bar sets the chart's BPM and GAP. It redraws as you
+  type and charts on **APPLY**. ×2 and ÷2 fix a tempo detected at double or
+  half speed, and **ALIGN TO NOTE** puts the selected note on a beat.
+  **RESET** goes back to songtrace's choice.
+- **Undoing an edit:** **REVERT** returns a line to the generated notes.
+
+Edits are kept apart from what the pipeline generates and laid back over it on
+every rebuild, so they survive all of them, collection-wide ones included. If
+you change a line's lyrics, its edit is set aside rather than applied to
+different words, and the sheet lists those for you to discard. Songs with edits
+are marked ✎ in the queue.
+
+## Rebuilding
+
+Select a song and its inputs come back up in the form; change them and press
+**REBUILD SONG**. Only the stages your change affects are rerun: fixing a line
+of lyrics realigns without downloading or separating again.
+
+A songtrace update can improve a stage without any of your inputs changing.
+**Restart from** on the rebuild form redoes a stage anyway, and
+**REBUILD ALL FROM…**, under CREATE SONG, does it for your whole collection:
+pick the stage the update improved, untick any songs to leave alone, and each
+song is rebuilt from there. Your edits are kept.
+
+Stages in order: `acquire`, `separate`, `transcribe`, `align`, `pitch`,
+`notes`, `grid`, `package`. Rebuilding from `notes` onward needs no GPU.
 
 ## Configuration
 
-Config is TOML. Resolution order:
+`config.toml` is TOML; `config.example.toml` lists every setting. songtrace
+reads, in order: `--config <path>`, the `SONGTRACE_CONFIG` environment variable
+(set to `/config/config.toml` in the container), then
+`~/.config/songtrace/config.toml`.
 
-1. `--config <path>` passed explicitly on the command line
+`whisper_api` chooses how to talk to Whisper: `"openai"` (the default) uses
-2. the `SONGTRACE_CONFIG` environment variable
+`POST {whisper_url}/audio/transcriptions`, and `"asr-webservice"` uses
-3. `~/.config/songtrace/config.toml`
+whisper-asr-webservice's `POST {whisper_url}/asr`. For the latter, give
+`whisper_url` without a `/v1` suffix; `whisper_model` and `whisper_api_key`
+are ignored, since that server's model is fixed when it starts.
 
-Copy the example and fill in your Whisper server:
+`creator` sets the `#CREATOR` header in every chart; it defaults to
-
+`songtrace`.
-```sh
-cp config.example.toml config.toml
-$EDITOR config.toml   # set whisper_url at minimum
-```
-
-See `config.example.toml` for every available field (`whisper_url`,
-`whisper_api_key`, `whisper_model`, `whisper_api`, `output_dir`, `work_dir`,
-`creator`, `demucs_model`, `crepe_model`, `device`, `bind_host`, `port`,
-`ytdlp_clients`, `ytdlp_cookies`, `ytdlp_impersonate`,
-`ytdlp_pot_provider_url`, `ytdlp_version`).
-
-`whisper_api` selects the wire format: `"openai"` (default) speaks the
-OpenAI-compatible `POST {whisper_url}/audio/transcriptions` API; `"asr-webservice"`
-speaks [onerahmet/openai-whisper-asr-webservice](https://github.com/ahmetoner/whisper-asr-webservice)'s
-`POST {whisper_url}/asr` API instead (point `whisper_url` at that server's base
-URL with no `/v1` suffix — `whisper_model`/`whisper_api_key` are unused for
-this backend, since the model is fixed by how that server was started).
-
-## Docker
-
-```sh
-cp config.example.toml config.toml
-$EDITOR config.toml
-docker compose up -d
-```
-
-The web UI is then at `http://localhost:8000`. Downloaded model weights
-(Demucs, wav2vec2, ...) are cached under `./data` on the host so they survive
-container recreation.
-
-Run the CLI inside the container instead of the web UI:
-
-```sh
-docker compose run --rm songtrace songtrace make \
-  --title "Song Title" --artist "Artist Name" \
-  --lyrics /data/lyrics.txt --youtube https://youtu.be/...
-```
-
-GPU passthrough requires the [NVIDIA Container
-Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/)
-installed on the host; `docker-compose.yml` requests one GPU via the
-`nvidia` device driver.
 
 ## Sharing the GPU
 
-`songtrace serve` is long-lived but only wants the GPU in bursts, so it hands
+Three stages load a model: `separate` (Demucs), `align` (wav2vec2) and `pitch`
-it back after every stage rather than holding it for the life of the process.
+(torchcrepe). Each releases its model and returns the memory to the driver as
-Three stages load a model — `separate` (Demucs), `align` (wav2vec2) and
+soon as it finishes, rather than holding it for the life of the server. This
-`pitch` (torchcrepe) — and each one's weights are dropped as soon as that
+matters most while `transcribe` waits on your Whisper server, which can take
-stage finishes, followed by `torch.cuda.empty_cache()` to return the
+minutes. The job log says how much each stage gave back:
-allocator's freed blocks to the driver instead of keeping them for reuse. This
-matters most between `separate` and `align`: `transcribe` sits between them
-waiting on your Whisper server, which can be minutes of a GPU nothing is
-using.
-
-Stages that gave memory back say so in the job log:
 
 ```
 [separate] released 3704 MiB of GPU memory
 ```
 
-The cost is reloading each model once per job instead of once per process,
+Reloading the models costs seconds per song. The CUDA context itself, a few
-which is seconds against stages that run for minutes. What this does not
+hundred MB, stays with the process; stop the container to free it.
-release is the CUDA context itself, a few hundred MB that belongs to the
-process until it exits; freeing that would mean running each stage in a
-subprocess. If you need every last MB back, stop the container between runs.
 
-## Bare-metal
+## YouTube download failures
+
+YouTube changes its defences every few weeks, so songtrace keeps `yt-dlp`
+current by itself. Once a day, and again when a download fails, it checks PyPI
+for a newer stable release, installs it under `/data/models/cache`, and tries
+it before the copy it shipped with. That copy is never modified and is always
+the fallback, so a broken release costs a retry, not the download. The job log
+names the `yt-dlp` version behind each attempt. `ytdlp_version` pins a release,
+or `"bundled"` turns the updates off.
+
+If downloads still fail with `HTTP Error 403: Forbidden`:
+
+1. **The network is rate-limited or flagged.** Datacenter and VPN ranges are
+   throttled hardest. Try another network before changing config.
+2. **A PO token is required.** If the log says so, run
+   [bgutil-ytdlp-pot-provider](https://github.com/Brainicism/bgutil-ytdlp-pot-provider)
+   alongside songtrace, install its plugin into the image, and point
+   `ytdlp_pot_provider_url` at it.
+3. **Other levers:** `ytdlp_clients` tries each listed YouTube player client
+   in turn. `ytdlp_cookies` takes a Netscape cookie jar for age-gated videos;
+   don't use one from an account you care about, as YouTube bans automated
+   access. `ytdlp_impersonate` presents a browser's TLS fingerprint (for
+   example `"chrome"`).
+
+To check the JavaScript runtime `yt-dlp` needs for YouTube is present, run
+`docker compose exec songtrace yt-dlp -v --simulate <url>`; the `JS runtimes:`
+line must not say `none`. A failed download keeps its working folder, so
+running the job again resumes it.
+
+## Command line
+
+The web UI is the main way to use songtrace, but every song can also be made
+from the command line, in the container or on a host with the requirements
+above and `ffmpeg`:
 
 ```sh
-pip install -e .
+docker compose run --rm songtrace songtrace make \
-songtrace make --help
+  --title "Song Title" --artist "Artist Name" \
+  --lyrics /data/lyrics.txt --youtube https://youtu.be/...
 ```
 
 ```
@@ -179,99 +199,28 @@ songtrace make --title TITLE --artist ARTIST --lyrics LYRICS_FILE \
   [--bpm BPM] [--force-from STAGE] [--config CONFIG]
 ```
 
-Run the web UI instead with `songtrace serve [--config CONFIG]`.
+`--force-from <stage>` redoes that stage and everything after it, reusing the
+cached output of the stages before. On a bare-metal host, install with
+`pip install -e .` and run the web UI with `songtrace serve`.
 
-## Re-running a stage
+## Development
-
-Each stage caches its output under `work_dir`. If a later stage needs
-another pass (e.g. you want to retry alignment with different lyrics, or
-pitch tracking looks off), re-run with `--force-from <stage>` to redo that
-stage and everything after it, reusing the cached output of everything
-before it:
-
-```sh
-songtrace make --title "..." --artist "..." --lyrics lyrics.txt \
-  --youtube https://youtu.be/... --force-from align
-```
-
-Valid stages, in pipeline order: `acquire`, `separate`, `transcribe`,
-`align`, `pitch`, `notes`, `grid`, `package`.
-
-In the web UI this is what selecting a queue entry does: its title, artist,
-lyrics and the rest come back up in the form, so you can adjust them and hit
-REBUILD SONG. Only the stages your edit actually invalidates are rerun —
-fixing a line of lyrics realigns without re-downloading or re-separating.
-CREATE SONG, at the top of the queue, clears the page to start a fresh one.
-
-That planning works off the fields you changed, which cannot see a change in
-songtrace itself: after an upgrade, a rebuild of an older song finds every
-stage cached and does nothing. **Restart from** on the rebuild form is the way
-to redo a stage anyway — pick `package` to repackage without re-downloading.
-It can only widen what is rerun, so an edit that invalidates an earlier stage
-still wins.
-
-**REBUILD ALL FROM…**, under CREATE SONG, does the same for every song at once:
-pick the stage an update improved and each song is rebuilt from there with the
-inputs it has now. Every song starts ticked; untick any already rebuilt by hand
-(each shows when it was last built). Songs still in progress are skipped, as
-are songs whose uploaded video is gone when the stage is `acquire`.
-
-## YouTube download failures
-
-`HTTP Error 403: Forbidden` from the acquire stage means YouTube served the
-page but refused the media URL. In rough order of likelihood:
-
-1. **yt-dlp is stale.** YouTube changes its defences every few weeks, so
-   songtrace keeps yt-dlp current itself. Once a day, and again whenever a
-   download fails outright, it checks PyPI for a newer stable release,
-   installs it under `$XDG_CACHE_HOME/songtrace/` (`/data/models/cache` in the
-   container), and tries it before the copy installed with songtrace. That
-   copy is never modified and is always the fallback, so a release that breaks
-   compatibility costs a retry rather than the download. The job log names
-   the yt-dlp version behind each attempt.
-
-   `ytdlp_version` overrides this: `"bundled"` uses only the installed copy,
-   and a release number such as `"2026.08.19"` pins that release instead of
-   the newest.
-
-2. **Missing JS runtime.** yt-dlp needs `yt-dlp-ejs` plus a JavaScript engine
-   to solve YouTube's player challenges; without them it silently falls back to
-   formats that 403. The `[default,curl-cffi,deno]` extras in `pyproject.toml`
-   cover this, for both the installed copy and the ones fetched at runtime.
-   Check with `docker compose exec songtrace yt-dlp -v --simulate <url>` — the debug
-   header must show a `JS runtimes:` line that is not `none`.
-
-3. **The IP is rate-limited or flagged.** Datacenter and VPN ranges get
-   throttled hardest. Try from another network before changing config.
-
-4. **A PO token is required.** If the log says so explicitly, run
-   [bgutil-ytdlp-pot-provider](https://github.com/Brainicism/bgutil-ytdlp-pot-provider)
-   as a sidecar, install its plugin into the image, and point
-   `ytdlp_pot_provider_url` at it.
-
-`ytdlp_clients` retries the download against each listed player client in
-turn, so a client that YouTube has broken is skipped automatically.
-`ytdlp_cookies` (a Netscape cookie jar, for age-gated videos) and
-`ytdlp_impersonate` (a browser TLS fingerprint, e.g. `"chrome"`) are the other
-two levers. Do not use a cookie jar from an account you care about — YouTube
-bans for automated access.
-
-Failed downloads leave the work dir intact, so re-running the job resumes
-rather than starting over.
-
-## Regenerating the web UI screenshot
-
-`scripts/screenshot_webui.py` seeds the real app with a fixed set of jobs
-covering every status and screenshots it to `docs/images/webui.png`. It uses
-`ffmpeg` to give the selected song a playable folder so the shot includes the
-player; without `ffmpeg` on `PATH` it still runs, just without it.
 
 ```sh
 pip install -e '.[dev]'
-playwright install chromium
+pytest
-python scripts/screenshot_webui.py
 ```
 
+`pytest` also runs the web UI's JavaScript tests in `tests/js` when Node is
+installed. Pull requests run the full suite in CI.
+
+songtrace is developed on a private forge and published to
+[GitHub](https://github.com/branderson/songtrace) as a read-only mirror; see
+`mirror/README.md`.
+
+`scripts/screenshot_webui.py` regenerates `docs/images/webui.png` from a
+seeded copy of the app (`playwright install chromium` first). With `ffmpeg` on
+`PATH`, the shot includes the player.
+
 ## License
 
 [AGPLv3](LICENSE)

config.example.toml

@@ -1,11 +1,11 @@
-whisper_url = "http://whisper.intranet:9000/v1"
+whisper_url = "http://whisper.example:9000/v1"
 # whisper_api_key = ""
 # whisper_model = "whisper-1"
 # whisper_api = "openai"  # or "asr-webservice" for onerahmet/openai-whisper-asr-webservice
-                          # (set whisper_url without the /v1 suffix, e.g. "http://whisper.intranet:9000")
+                          # (set whisper_url without the /v1 suffix, e.g. "http://whisper.example:9000")
 output_dir = "/data/output"
 work_dir = "/data/work"
-creator = "brad"
+# creator = "Your Name"  # the #CREATOR header in every chart
 # demucs_model = "htdemucs_ft"
 # crepe_model = "full"
 # device = "cuda"

mirror/README.md

@@ -0,0 +1,25 @@
+# GitHub mirror
+
+songtrace is developed on a private forge. `github-mirror.sh` publishes its
+`main` and the tags on `main` to
+[github.com/branderson/songtrace](https://github.com/branderson/songtrace),
+hourly, from a host that can push to GitHub. Issues and pull requests are off
+there.
+
+The mirror's `main` is the forge's plus one commit, which adds a read-only notice
+under the README's title. Every forge commit is in the mirror unchanged, and
+each sync fast-forwards. If the README loses its `# ` title, the sync stops
+instead of publishing without the notice.
+
+## Setup
+
+1. Set up the timer as described at the top of
+   `songtrace-github-mirror.service`. Give it a deploy key with write access to
+   the GitHub repository.
+2. Run `github-mirror.sh --dry-run` to build what a sync would publish without
+   pushing it.
+3. Once the first sync has pushed, run `github-mirror.sh --settings` (needs
+   `gh`). It turns off issues, pull requests, projects, wiki and discussions.
+
+The timer runs a copy of the script, not the checkout's. Reinstall it after
+reviewing a change to `mirror/`.

mirror/github-mirror.sh

@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+# Publishes main and its release tags to a read-only mirror on GitHub.
+#
+#   MIRROR_SOURCE=<forge clone URL> mirror/github-mirror.sh    sync
+#   ... mirror/github-mirror.sh --dry-run                      build and print, push nothing
+#   mirror/github-mirror.sh --settings                         configure the GitHub repo (needs gh)
+#   mirror/github-mirror.sh --transform DIR                    rewrite a checkout in place
+#
+# Runs on a host that can push to GitHub, never in a dejima session: a sandbox
+# may push only to its own forge.
+#
+# The GitHub copy is not byte-for-byte. Its main is one extra commit on top of
+# the forge's main, whose tree is the forge's with the rewrites in transform()
+# applied, and whose parents are the forge commit and the previous mirror
+# commit. So every forge SHA is in the mirror unchanged, and each sync is a
+# fast-forward for anyone tracking it. Tags are pushed as they are, pointing at
+# the untransformed forge commits.
+#
+# Environment:
+#   MIRROR_SOURCE  forge clone URL (required to sync)
+#   MIRROR_REPO    GitHub repository, default branderson/songtrace
+#   MIRROR_TARGET  push URL, default git@github.com:$MIRROR_REPO.git
+#   MIRROR_STATE   working bare repository, default ~/.cache/songtrace-github-mirror.git
+set -euo pipefail
+
+MIRROR_REPO="${MIRROR_REPO:-branderson/songtrace}"
+MIRROR_TARGET="${MIRROR_TARGET:-git@github.com:$MIRROR_REPO.git}"
+MIRROR_STATE="${MIRROR_STATE:-${XDG_CACHE_HOME:-$HOME/.cache}/songtrace-github-mirror.git}"
+
+say() { echo "github-mirror: $*"; }
+die() { say "$*" >&2; exit 1; }
+
+WORK=""
+cleanup() {
+    [ -n "$WORK" ] || return 0
+    git -C "$MIRROR_STATE" worktree remove --force "$WORK" 2>/dev/null || true
+    rm -rf "$WORK"
+}
+
+# Rewrites a checkout for GitHub. Each rewrite is checked, so a README that
+# drifted away from what this expects stops the mirror instead of publishing
+# the forge's version. Running it twice changes nothing.
+transform() {
+    local readme="$1/README.md"
+    # perl -i would write a symlink's target back as the README, publishing it.
+    [ -f "$readme" ] && [ ! -L "$readme" ] || die "README.md is missing or a symlink in $1"
+
+    # The first thing a visitor to a mirror does is open an issue or a pull
+    # request. Under the title.
+    grep -q '^> \*\*This is a read-only mirror' "$readme" || perl -0pi -e '
+        s{^(# [^\n]*)\n}{$1\n\n> **This is a read-only mirror.** songtrace is developed on a private forge,\n> and issues and pull requests are not accepted here.\n}m
+    ' "$readme"
+    grep -q '^> \*\*This is a read-only mirror' "$readme" \
+        || die "README.md: no title to put the mirror notice under"
+}
+
+# Issues, pull requests, wiki and the like off. Settings rather than content,
+# so a sync does not reapply them; run once after creating the repository.
+settings() {
+    command -v gh >/dev/null || die "--settings needs the gh CLI"
+    gh api -X PATCH "repos/$MIRROR_REPO" --silent \
+        -F has_issues=false -F has_pull_requests=false -F has_projects=false \
+        -F has_wiki=false -F has_discussions=false
+}
+
+sync() {
+    [ -n "${MIRROR_SOURCE:-}" ] || die "MIRROR_SOURCE is not set"
+    local git=(git -C "$MIRROR_STATE")
+    trap cleanup EXIT
+    [ -d "$MIRROR_STATE" ] || git init -q --bare "$MIRROR_STATE"
+
+    "${git[@]}" fetch -q --no-tags --prune "$MIRROR_SOURCE" \
+        '+refs/heads/main:refs/source/main' '+refs/tags/*:refs/source/tags/*'
+    local src prev=""
+    src="$("${git[@]}" rev-parse refs/source/main)"
+
+    # An absent main (exit 2) is a first run; anything else is a real failure.
+    local rc=0
+    git ls-remote -q --exit-code "$MIRROR_TARGET" refs/heads/main >/dev/null || rc=$?
+    if [ "$rc" -eq 0 ]; then
+        "${git[@]}" fetch -q --no-tags "$MIRROR_TARGET" '+refs/heads/main:refs/mirror/main'
+        prev="$("${git[@]}" rev-parse refs/mirror/main)"
+    elif [ "$rc" -ne 2 ]; then
+        die "cannot read $MIRROR_TARGET"
+    fi
+
+    WORK="$(mktemp -d)"
+    "${git[@]}" worktree add -q --detach "$WORK" "$src"
+    transform "$WORK"
+    git -C "$WORK" add -A
+    local tree
+    tree="$(git -C "$WORK" write-tree)"
+
+    if [ -n "$prev" ] && [ "$("${git[@]}" rev-parse "$prev^1")" = "$src" ] \
+        && [ "$("${git[@]}" rev-parse "$prev^{tree}")" = "$tree" ]; then
+        say "already at ${src:0:12}"
+    else
+        # The forge commit's committer and date, so the same input always
+        # makes the same commit.
+        local name email date commit
+        IFS=$'\t' read -r name email date < <("${git[@]}" log -1 --format='%cn%x09%ce%x09%cI' "$src")
+        commit="$(GIT_AUTHOR_NAME="$name" GIT_AUTHOR_EMAIL="$email" GIT_AUTHOR_DATE="$date" \
+            GIT_COMMITTER_NAME="$name" GIT_COMMITTER_EMAIL="$email" GIT_COMMITTER_DATE="$date" \
+            "${git[@]}" commit-tree "$tree" -p "$src" ${prev:+-p "$prev"} \
+            -m "Mirror ${src:0:12} for GitHub" \
+            -m "README header added by mirror/github-mirror.sh.")"
+        "${git[@]}" update-ref refs/mirror/next "$commit"
+        [ -n "$DRY_RUN" ] || "${git[@]}" push -q "$MIRROR_TARGET" "$commit:refs/heads/main"
+        say "main is ${src:0:12} as ${commit:0:12}"
+    fi
+
+    # Only tags on main: one on another branch would publish unreleased work.
+    local ref
+    local refspecs=()
+    while IFS= read -r ref; do
+        if "${git[@]}" merge-base --is-ancestor "$ref" "$src"; then
+            refspecs+=("$ref:refs/tags/${ref#refs/source/tags/}")
+            say "tag ${ref#refs/source/tags/}"
+        fi
+    done < <("${git[@]}" for-each-ref --format='%(refname)' refs/source/tags/)
+    [ "${#refspecs[@]}" -eq 0 ] || [ -n "$DRY_RUN" ] \
+        || "${git[@]}" push -q "$MIRROR_TARGET" "${refspecs[@]}"
+}
+
+DRY_RUN=""
+case "${1:-}" in
+    "")          sync ;;
+    --dry-run)   DRY_RUN=1; sync ;;
+    --settings)  settings ;;
+    --transform) [ -n "${2:-}" ] || die "--transform needs a directory"; transform "$2" ;;
+    *)           sed -n '4,7p' "$0" >&2; exit 2 ;;
+esac

mirror/songtrace-github-mirror.service

@@ -0,0 +1,26 @@
+# User unit for the GitHub mirror, on a host that can push to GitHub. Never in
+# a dejima session: a sandbox may push only to its own forge.
+#
+#   install -m 755 mirror/github-mirror.sh ~/.local/bin/songtrace-github-mirror
+#   cp mirror/songtrace-github-mirror.service mirror/songtrace-github-mirror.timer ~/.config/systemd/user/
+#   systemctl --user daemon-reload
+#   systemctl --user enable --now songtrace-github-mirror.timer
+#
+# A copy, not a link into the checkout: the timer holds a key that can push to
+# GitHub, and a session working in this checkout can edit the script.
+# Reinstall it after reviewing an update.
+#
+# ~/.config/songtrace/github-mirror.env sets MIRROR_SOURCE, the forge
+# clone URL, and whatever else mirror/github-mirror.sh documents. The GitHub
+# key, a deploy key with write access to the repository, belongs behind an ssh
+# Host alias named in MIRROR_TARGET, so it is not also offered to the forge:
+#
+#   MIRROR_SOURCE=git@forge.example.com:owner/songtrace.git
+#   MIRROR_TARGET=git@github-songtrace:branderson/songtrace.git
+[Unit]
+Description=Mirror songtrace's main and release tags to GitHub
+
+[Service]
+Type=oneshot
+EnvironmentFile=%h/.config/songtrace/github-mirror.env
+ExecStart=%h/.local/bin/songtrace-github-mirror

mirror/songtrace-github-mirror.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Mirror songtrace's main and release tags to GitHub, hourly
+
+[Timer]
+OnCalendar=hourly
+Persistent=true
+
+[Install]
+WantedBy=timers.target

scripts/screenshot_webui.py

@@ -15,11 +15,14 @@ import time
 from pathlib import Path
 
 import httpx
+import numpy as np
 import uvicorn
 from playwright.sync_api import sync_playwright
 
 from songtrace.config import Config
+from songtrace.grid import to_ultrastar
 from songtrace.pipeline import PipelineError, PipelineResult, song_slug
+from songtrace.ultrastar import serialize_ultrastar
 from songtrace.web import create_app
 
 REPO_ROOT = Path(__file__).resolve().parent.parent
@@ -128,13 +131,29 @@ def make_runner():
             if "review" in extra:
                 work = cfg.work_dir / song_slug(inp)
                 work.mkdir(parents=True, exist_ok=True)
+                # one note per word in the seeded lines, so each sings its word
                 notes = [
-                    {**n, "duration": n["end"] - n["start"], "text": "~",
+                    {**n, "duration": n["end"] - n["start"],
-                     "line_index": line["index"]}
+                     "text": (" " if k else "") + w["text"], "line_index": line["index"]}
-                    for line in extra["review"]["lines"] for n in line["notes"]
+                    for line in extra["review"]["lines"]
+                    for k, (n, w) in enumerate(zip(line["notes"], line["words"]))
                 ]
+                sung = [line for line in extra["review"]["lines"] if line["notes"]]
+                breaks = [(a["notes"][-1]["end"] + b["notes"][0]["start"]) / 2
+                          for a, b in zip(sung, sung[1:])]
                 (work / "notes_sec.json").write_text(json.dumps({
-                    "notes": notes, "breaks": [], "warnings": extra["warnings"]}))
+                    "notes": notes, "breaks": breaks, "warnings": extra["warnings"]}))
+                # a chart for the beat grid, and a sung pitch with a little
+                # vibrato for the curve over the notes
+                (work / "song.txt").write_text(serialize_ultrastar(to_ultrastar(
+                    notes, breaks, 240.0, [("TITLE", inp.title)])), encoding="utf-8")
+                times = np.arange(0, 60, 0.01)
+                f0, conf = np.zeros_like(times), np.zeros_like(times)
+                for n in notes:
+                    m = (times >= n["start"] + 0.02) & (times < n["start"] + n["duration"] - 0.02)
+                    semis = n["pitch"] + 0.2 * np.sin(2 * np.pi * 5.5 * times[m])
+                    f0[m], conf[m] = 440 * 2 ** ((semis - 9) / 12), 0.9
+                np.savez(work / "pitch.npz", times=times, f0=f0, conf=conf)
                 (work / "align_meta.json").write_text(json.dumps({
                     "dropped_lines": [line["index"] for line in extra["review"]["lines"]
                                       if line["dropped"]],
@@ -230,13 +249,13 @@ def capture(base_url: str, with_player: bool) -> None:
             # current word colouring rather than an untouched transport
             page.evaluate("document.getElementById('player-media').currentTime = 11.6")
             page.wait_for_timeout(400)
-            # scrolled to the review: the player pins to the top of #main and
+        page.wait_for_selector(".sheet-curve polyline")
-            # the tracked lines sit under it, which is the panel in use
+        # scrolled to the review, the panel in use, with the note sheet whole
-            page.evaluate("""() => {
+        page.evaluate("""() => {
-              const main = document.getElementById('main');
+          const main = document.getElementById('main');
-              main.scrollTo(0, document.getElementById('review').offsetTop - 12);
+          main.scrollTo(0, document.getElementById('review').offsetTop - 12);
-            }""")
+        }""")
-            page.wait_for_timeout(400)
+        page.wait_for_timeout(400)
         OUT_IMAGE.parent.mkdir(parents=True, exist_ok=True)
         page.screenshot(path=str(OUT_IMAGE))
         browser.close()

src/songtrace/static/index.html

@@ -623,6 +623,11 @@ button { cursor: pointer; }
 }
 .lineage:hover { border-bottom-color: var(--aqua); }
 .job-mark { grid-area: 1 / 2; justify-self: end; font-size: 0.8rem; color: var(--text-muted); }
+.job-edits { grid-area: 2 / 2; justify-self: end; display: flex; gap: 6px; font-size: 0.72rem; }
+.job-edits-kept { color: var(--aqua); }
+.job-edits-stale { color: var(--warn); }
+.rebuild-all-kept { color: var(--aqua); }
+.rebuild-all-aside { color: var(--warn); }
 
 #detail {
   margin: 13px 0 0;
@@ -744,7 +749,8 @@ button { cursor: pointer; }
   <form id="rebuild-all-form" method="dialog">
     <h2 id="rebuild-all-heading">REBUILD ALL SONGS</h2>
     <p>Reruns the songs ticked below from the chosen stage onwards with the
-      inputs they have now, for when a songtrace update improves that stage.</p>
+      inputs they have now, for when a songtrace update improves that stage.
+      Lines edited by hand are kept.</p>
     <label class="field">
       <span>Restart from</span>
       <select name="force_from" required></select>
@@ -991,6 +997,18 @@ function rebuildAllRow(song) {
     undefined, { dateStyle: "medium", timeStyle: "short" });
   meta.textContent = busy ? "in progress, skipped"
     : `${song.status === "failed" ? "failed" : song.rebuilt_from ? "rebuilt" : "built"} ${when || ""}`.trim();
+  if (song.edited_lines) {
+    const kept = document.createElement("span");
+    kept.className = "rebuild-all-kept";
+    kept.textContent = ` · ${song.edited_lines} edited line${song.edited_lines === 1 ? "" : "s"} kept`;
+    meta.append(kept);
+  }
+  if (song.stale_edits) {
+    const aside = document.createElement("span");
+    aside.className = "rebuild-all-aside";
+    aside.textContent = ` · ${song.stale_edits} set aside`;
+    meta.append(aside);
+  }
   row.append(box, name, meta);
   return row;
 }
@@ -1066,7 +1084,10 @@ function jobCard(job) {
   markLabel.className = "sr-only";
   mark.append(markGlyph, markLabel);
 
-  card.append(title, artist, stateLine, time, mark);
+  const edits = document.createElement("span");
+  edits.className = "job-edits";
+
+  card.append(title, artist, stateLine, time, mark, edits);
   card.addEventListener("click", () => selectJob(job.id));
   paintCard(card, job);
   return card;
@@ -1074,7 +1095,7 @@ function jobCard(job) {
 
 function paintCard(card, job) {
   const state = STATE[job.status] ?? { glyph: "·", cls: "st-queued" };
-  const [title, artist, stateLine, time, mark] = card.children;
+  const [title, artist, stateLine, time, mark, edits] = card.children;
   card.className = `job ${state.cls}`;
   if (job.id === selectedId) card.setAttribute("aria-current", "true");
   else card.removeAttribute("aria-current");
@@ -1085,6 +1106,33 @@ function paintCard(card, job) {
   time.textContent = elapsed(job);
   mark.children[0].textContent = job.rebuilt_from ? "↻" : "";
   mark.children[1].textContent = job.rebuilt_from ? " rebuilt" : "";
+  paintEdits(edits, job);
+}
+
+// hand edits a rebuild keeps, and any set aside because the lyrics changed
+function paintEdits(node, job) {
+  const parts = [];
+  if (job.edited_lines) parts.push(["job-edits-kept", `✎ ${job.edited_lines}`]);
+  if (job.stale_edits) parts.push(["job-edits-stale", `⚠ ${job.stale_edits}`]);
+  const label = [
+    job.edited_lines ? `${job.edited_lines} edited line${job.edited_lines === 1 ? "" : "s"}` : "",
+    job.stale_edits ? `${job.stale_edits} set aside` : "",
+  ].filter(Boolean).join(", ");
+  if (node.title === label) return;
+  node.title = label;
+  node.replaceChildren(...parts.map(([cls, text]) => {
+    const span = document.createElement("span");
+    span.className = cls;
+    span.setAttribute("aria-hidden", "true");
+    span.textContent = text;
+    return span;
+  }));
+  if (label) {
+    const sr = document.createElement("span");
+    sr.className = "sr-only";
+    sr.textContent = label;
+    node.append(sr);
+  }
 }
 
 // patched in place rather than replaced wholesale: detaching a card would blur

src/songtrace/static/sheet.js

@@ -10,6 +10,7 @@ const BLACK = new Set([1, 3, 6, 8, 10]);
 const GUTTER = 46;        // px of note names left of the grid
 const PLOT_HEIGHT = 300;  // px; rows divide this between the pitches shown
 const CONTEXT = 0.75;     // seconds shown either side of the line
+const BREAK_REACH = 3;    // seconds past the line the page grows to show a break
 const EDGE = 6;           // px from a note's end that grabs it for resizing
 const KINDS = { "*": "golden", F: "free", R: "rap", G: "golden-rap" };
 const KEYS_OPEN = "songtrace.sheet-keys";
@@ -298,11 +299,13 @@ export function createSheet(root, { seek, playLine, onSaved }) {
     const span = ts.length ? { start: Math.min(...ts), end: Math.max(...ts) } : lineSpan(line);
     const range = pitchRange(notes().map(n => n.pitch));
     const want = { t0: span.start - CONTEXT, t1: span.end + CONTEXT, ...range };
-    // the breaks either side are part of the line's page, so they stay on it
+    // the breaks either side are part of the line's page, so it grows to
+    // show them, but not across a long instrumental: one further away is
+    // pinned to the edge instead (see shownBreakTime)
     for (const slot of breakSlots()) {
       const t = breakTime(slot);
-      want.t0 = Math.min(want.t0, t - 0.3);
+      want.t0 = Math.min(want.t0, Math.max(t - 0.3, span.start - BREAK_REACH));
-      want.t1 = Math.max(want.t1, t + 0.3);
+      want.t1 = Math.max(want.t1, Math.min(t + 0.3, span.end + BREAK_REACH));
     }
     layout = reset || !layout ? want : {
       t0: Math.min(layout.t0, want.t0), t1: Math.max(layout.t1, want.t1),
@@ -413,6 +416,15 @@ export function createSheet(root, { seek, playLine, onSaved }) {
     return Math.min(Math.max(t, slot.lo), Math.max(slot.hi, slot.lo));
   }
 
+  // where a break is drawn and grabbed: on the page, pinned to its edge
+  // when it lies beyond it
+  function shownBreakTime(slot) {
+    const edge = (layout.t1 - layout.t0) * 0.003;   // a few pixels in
+    const t = breakTime(slot);
+    const shown = Math.min(Math.max(t, layout.t0 + edge), layout.t1 - edge);
+    return { t: shown, pinned: shown !== t ? (t < shown ? "left" : "right") : null };
+  }
+
   const undo = () => { if (hist?.canUndo) { hist.undo(); fit(false); render(); } };
   const redo = () => { if (hist?.canRedo) { hist.redo(); fit(false); render(); } };
   const one = () => (selected.size === 1 ? [...selected][0] : null);
@@ -617,14 +629,16 @@ export function createSheet(root, { seek, playLine, onSaved }) {
 
     const marks = el("g", { class: "sheet-breaks" }, svg);
     for (const slot of breakSlots()) {
-      const t = breakTime(slot);
+      const { t, pinned } = shownBreakTime(slot);
       const value = hist.present[slot.which];
       const byHand = value !== null &&
         (value !== baseline[slot.which] || review.breaks?.find(b => b.line === slot.line)?.moved);
       const cls = `sheet-break${byHand ? " sheet-break-moved" : ""}`;
       el("line", { x1: x(t), x2: x(t), y1: 0, y2: PLOT_HEIGHT, class: cls }, marks);
-      el("text", { x: x(t) + 3, y: 11, class: `${cls} sheet-break-label` }, marks)
+      const label = el("text", { x: pinned === "right" ? x(t) - 3 : x(t) + 3, y: 11,
-        .textContent = "↵";
+                                 "text-anchor": pinned === "right" ? "end" : "start",
+                                 class: `${cls} sheet-break-label` }, marks);
+      label.textContent = pinned === "right" ? "↵ →" : pinned === "left" ? "← ↵" : "↵";
     }
 
     syllables.replaceChildren(...notes().map(n => {
@@ -708,7 +722,7 @@ export function createSheet(root, { seek, playLine, onSaved }) {
     // by geometry, not the event's target: a render replaces the elements
     // under the pointer, and the topmost note drawn wins
     if (canEdit()) {
-      const brk = breakSlots().find(s => Math.abs(px - view.x(breakTime(s))) <= 5);
+      const brk = breakSlots().find(s => Math.abs(px - view.x(shownBreakTime(s).t)) <= 5);
       if (brk) return { px, py, note: null, brk, mode: "break" };
     }
     const pitch = view.pitch(py);

src/songtrace/web.py

@@ -342,11 +342,21 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
             position = {job_id: n for n, job_id in enumerate(order)}
             return sorted(latest.values(), key=lambda j: position[j.id], reverse=True)
 
+    def _edit_counts(job: Job) -> dict:
+        """How much of the song was edited by hand, from its edits file."""
+        path = cfg.work_dir / song_slug(job.inp) / edits.EDITS_FILE
+        try:
+            data = json.loads(path.read_text())
+        except (OSError, ValueError):
+            return {"edited_lines": 0, "stale_edits": 0}
+        return {"edited_lines": len(data.get("lines", {})),
+                "stale_edits": len(data.get("stale", []))}
+
     @app.get("/api/rebuild-all")
     def rebuild_all_candidates():
         return [{"id": j.id, "title": j.title, "artist": j.artist,
                  "status": j.status, "rebuilt_from": j.rebuilt_from,
-                 "finished_at": j.finished_at}
+                 "finished_at": j.finished_at, **_edit_counts(j)}
                 for j in _current_songs()]
 
     @app.post("/api/rebuild-all", status_code=201)
@@ -382,11 +392,16 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
 
     @app.get("/api/jobs")
     def list_jobs():
+        # edits belong to the song as it is now, so only the job standing for
+        # it carries them; the runs it replaced would claim them twice
+        counts = {j.id: _edit_counts(j) for j in _current_songs()}
+        none = {"edited_lines": 0, "stale_edits": 0}
         with lock:
             return [{"id": j.id, "title": j.title, "artist": j.artist,
                       "status": j.status, "stage": j.stage,
                       "created_at": j.created_at, "started_at": j.started_at,
-                      "finished_at": j.finished_at, "rebuilt_from": j.rebuilt_from}
+                      "finished_at": j.finished_at, "rebuilt_from": j.rebuilt_from,
+                      **counts.get(j.id, none)}
                      for j in (jobs[i] for i in reversed(order))]
 
     @app.get("/api/jobs/{job_id}")

tests/test_web.py

@@ -2074,3 +2074,40 @@ def test_a_bad_break_is_refused(tmp_path):
     job_id = make_charted_job(client)
     for body in ({"0": 1.0}, {"9": 1.0}, {"1": -2}, {"1": "x"}):
         assert client.put(f"/api/jobs/{job_id}/breaks", json=body).status_code == 400
+
+
+def _listed(client, job_id):
+    return next(j for j in client.get("/api/jobs").json() if j["id"] == job_id)
+
+
+def test_the_queue_reports_a_songs_edited_lines(tmp_path):
+    client, _ = make_client(tmp_path, chart_runner)
+    job_id = make_charted_job(client)
+    assert _listed(client, job_id)["edited_lines"] == 0
+    client.put(f"/api/jobs/{job_id}/lines/1", json=EDIT)
+    assert _listed(client, job_id)["edited_lines"] == 1
+    assert _listed(client, job_id)["stale_edits"] == 0
+
+
+def test_edit_counts_follow_the_song_to_its_latest_run(tmp_path):
+    client, _ = make_client(tmp_path, chart_runner)
+    first = make_charted_job(client)
+    client.put(f"/api/jobs/{first}/lines/1", json=EDIT)
+    rerun = client.post(f"/api/jobs/{first}/rerun").json()["id"]
+    wait_status(client, rerun, "done")
+    assert _listed(client, rerun)["edited_lines"] == 1
+    assert _listed(client, first)["edited_lines"] == 0
+    candidates = client.get("/api/rebuild-all").json()
+    assert [(c["id"], c["edited_lines"]) for c in candidates] == [(rerun, 1)]
+
+
+def test_the_queue_reports_edits_set_aside(tmp_path):
+    client, _ = make_client(tmp_path, chart_runner)
+    first = make_charted_job(client)
+    client.put(f"/api/jobs/{first}/lines/1", json=EDIT)
+    second = make_charted_job(client, lyrics="one\nthree more")
+    # the next save looks at the edits against the new lyrics, as a rebuild does
+    client.put(f"/api/jobs/{second}/lines/0", json={"notes": [
+        {"start": 1.0, "duration": 0.5, "pitch": 0, "text": "one"}]})
+    listed = _listed(client, second)
+    assert (listed["edited_lines"], listed["stale_edits"]) == (1, 1)