Compare commits
16
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
22c7f6d03f | ||
|
|
7c4359eb39 | ||
|
|
ba8bf9f4af | ||
|
|
be67e01943 | ||
|
|
b5850042be | ||
|
|
55e8297672 | ||
|
|
a7622c2d01 | ||
|
|
c489dd5742 | ||
|
|
f1b967b487 | ||
|
|
e141cc5773 | ||
|
|
92948b2ea7 | ||
|
|
c8c3d941ca | ||
|
|
7b8fce5b6e | ||
|
|
cb10400a21 | ||
|
|
98e4f9532f | ||
|
|
77bebcf9e1 |
@@ -34,6 +34,27 @@ printing the low-confidence regions it found (e.g.
|
||||
editor such as [Yass](https://www.yass-along.com/) or
|
||||
[USDX](https://usdx.eu/) and touch those spots up before playing.
|
||||
|
||||
## Reviewing a song
|
||||
|
||||
The web UI plays the finished song folder back against the review, so you can
|
||||
check the timing without loading it into a game first. Pick a song in the
|
||||
queue and the review panel gets the packaged video with a transport above the
|
||||
lyrics; as it plays, the line being sung is highlighted, words light up as
|
||||
they are reached, and a playhead sweeps each line against its notes. Click any
|
||||
line or word to play from there.
|
||||
|
||||
Playback is modelled on the game: the `#MP3` is the audio and the clock, and
|
||||
the `#VIDEO` is muted background slaved to it. It also uses the same seconds
|
||||
the notes were generated from, so a word that lights up late in the browser is
|
||||
late in the game too.
|
||||
|
||||
The packaged video has its audio track stripped for the same reason. It is the
|
||||
same recording as the `#MP3`, so a player that does not mute the video plays
|
||||
both at once, a few frames apart, which is heard as an echo. Songs built
|
||||
before this change still have it — check with `ffprobe <song>.mp4` for an
|
||||
audio stream, and rebuild with **Restart from** set to `package` to clear it
|
||||
without re-downloading.
|
||||
|
||||
## Configuration
|
||||
|
||||
Config is TOML. Resolution order:
|
||||
@@ -51,7 +72,9 @@ $EDITOR config.toml # set whisper_url at minimum
|
||||
|
||||
See `config.example.toml` for every available field (`whisper_url`,
|
||||
`whisper_api_key`, `whisper_model`, `whisper_api`, `output_dir`, `work_dir`,
|
||||
`creator`, `demucs_model`, `crepe_model`, `device`, `bind_host`, `port`).
|
||||
`creator`, `demucs_model`, `crepe_model`, `device`, `bind_host`, `port`,
|
||||
`ytdlp_clients`, `ytdlp_cookies`, `ytdlp_impersonate`,
|
||||
`ytdlp_pot_provider_url`).
|
||||
|
||||
`whisper_api` selects the wire format: `"openai"` (default) speaks the
|
||||
OpenAI-compatible `POST {whisper_url}/audio/transcriptions` API; `"asr-webservice"`
|
||||
@@ -117,10 +140,65 @@ songtrace make --title "..." --artist "..." --lyrics lyrics.txt \
|
||||
Valid stages, in pipeline order: `acquire`, `separate`, `transcribe`,
|
||||
`align`, `pitch`, `notes`, `grid`, `package`.
|
||||
|
||||
In the web UI this is what selecting a queue entry does: its title, artist,
|
||||
lyrics and the rest come back up in the form, so you can adjust them and hit
|
||||
REBUILD SONG. Only the stages your edit actually invalidates are rerun —
|
||||
fixing a line of lyrics realigns without re-downloading or re-separating.
|
||||
CREATE SONG, at the top of the queue, clears the page to start a fresh one.
|
||||
|
||||
That planning works off the fields you changed, which cannot see a change in
|
||||
songtrace itself: after an upgrade, a rebuild of an older song finds every
|
||||
stage cached and does nothing. **Restart from** on the rebuild form is the way
|
||||
to redo a stage anyway — pick `package` to repackage without re-downloading.
|
||||
It can only widen what is rerun, so an edit that invalidates an earlier stage
|
||||
still wins.
|
||||
|
||||
## YouTube download failures
|
||||
|
||||
`HTTP Error 403: Forbidden` from the acquire stage means YouTube served the
|
||||
page but refused the media URL. In rough order of likelihood:
|
||||
|
||||
1. **yt-dlp is stale.** YouTube changes its defences every few weeks and the
|
||||
version baked into the image does not. Hotfix a running container, then
|
||||
rebuild so it sticks:
|
||||
|
||||
```sh
|
||||
docker compose exec songtrace pip install -U 'yt-dlp[default,curl-cffi,deno]'
|
||||
docker compose restart songtrace
|
||||
docker compose build --pull songtrace # permanent
|
||||
```
|
||||
|
||||
2. **Missing JS runtime.** yt-dlp needs `yt-dlp-ejs` plus a JavaScript engine
|
||||
to solve YouTube's player challenges; without them it silently falls back to
|
||||
formats that 403. The `[default,curl-cffi,deno]` extras in `pyproject.toml`
|
||||
cover this. Check with
|
||||
`docker compose exec songtrace yt-dlp -v --simulate <url>` — the debug
|
||||
header must show a `JS runtimes:` line that is not `none`.
|
||||
|
||||
3. **The IP is rate-limited or flagged.** Datacenter and VPN ranges get
|
||||
throttled hardest. Try from another network before changing config.
|
||||
|
||||
4. **A PO token is required.** If the log says so explicitly, run
|
||||
[bgutil-ytdlp-pot-provider](https://github.com/Brainicism/bgutil-ytdlp-pot-provider)
|
||||
as a sidecar, install its plugin into the image, and point
|
||||
`ytdlp_pot_provider_url` at it.
|
||||
|
||||
`ytdlp_clients` retries the download against each listed player client in
|
||||
turn, so a client that YouTube has broken is skipped automatically.
|
||||
`ytdlp_cookies` (a Netscape cookie jar, for age-gated videos) and
|
||||
`ytdlp_impersonate` (a browser TLS fingerprint, e.g. `"chrome"`) are the other
|
||||
two levers. Do not use a cookie jar from an account you care about — YouTube
|
||||
bans for automated access.
|
||||
|
||||
Failed downloads leave the work dir intact, so re-running the job resumes
|
||||
rather than starting over.
|
||||
|
||||
## Regenerating the web UI screenshot
|
||||
|
||||
`scripts/screenshot_webui.py` seeds the real app with a fixed set of jobs
|
||||
covering every status and screenshots it to `docs/images/webui.png`:
|
||||
covering every status and screenshots it to `docs/images/webui.png`. It uses
|
||||
`ffmpeg` to give the selected song a playable folder so the shot includes the
|
||||
player; without `ffmpeg` on `PATH` it still runs, just without it.
|
||||
|
||||
```sh
|
||||
pip install -e '.[dev]'
|
||||
|
||||
@@ -11,3 +11,10 @@ creator = "brad"
|
||||
# device = "cuda"
|
||||
bind_host = "0.0.0.0"
|
||||
# port = 8000
|
||||
|
||||
# YouTube downloads. Each player client is tried in order until one works;
|
||||
# "default" means whatever yt-dlp currently picks for itself.
|
||||
# ytdlp_clients = ["default", "tv"]
|
||||
# ytdlp_cookies = "/config/youtube-cookies.txt" # Netscape cookie jar
|
||||
# ytdlp_impersonate = "chrome" # TLS fingerprint to present
|
||||
# ytdlp_pot_provider_url = "http://bgutil-provider:4416"
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 77 KiB After Width: | Height: | Size: 84 KiB |
+6
-1
@@ -5,7 +5,12 @@ description = "Generate UltraStar/Melody Mania karaoke songs from a video and re
|
||||
requires-python = ">=3.11"
|
||||
license = "AGPL-3.0-or-later"
|
||||
dependencies = [
|
||||
"yt-dlp",
|
||||
# yt-dlp declares no runtime deps of its own; the extras are what make
|
||||
# YouTube work. "default" pulls yt-dlp-ejs (JS challenge solving) and the
|
||||
# requests/urllib3 handlers, "deno" ships the JS runtime that executes it,
|
||||
# and "curl-cffi" provides the --impersonate targets. Without them YouTube
|
||||
# extraction degrades to HTTP 403 on the media URLs.
|
||||
"yt-dlp[default,curl-cffi,deno]",
|
||||
"demucs",
|
||||
"torch",
|
||||
"torchaudio",
|
||||
|
||||
@@ -7,6 +7,8 @@ One-time setup: pip install -e '.[dev]' && playwright install chromium
|
||||
Run: python scripts/screenshot_webui.py
|
||||
"""
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
@@ -110,6 +112,7 @@ SONGS = [
|
||||
("Vampire in the Corner", "Magdalena Bay", "queued", {}),
|
||||
]
|
||||
SELECTED = "Greensleeves"
|
||||
REVIEW_HEADING = "REVIEW"
|
||||
|
||||
running_event = threading.Event()
|
||||
|
||||
@@ -157,14 +160,42 @@ def make_runner():
|
||||
return runner
|
||||
|
||||
|
||||
def start_server(tmp_path: Path) -> str:
|
||||
def start_server(tmp_path: Path) -> tuple[str, Config]:
|
||||
cfg = Config(whisper_url="http://localhost:9000/v1", output_dir=tmp_path / "out",
|
||||
work_dir=tmp_path / "work", bind_host="127.0.0.1", port=8811)
|
||||
app = create_app(cfg, runner=make_runner())
|
||||
server = uvicorn.Server(uvicorn.Config(app, host=cfg.bind_host, port=cfg.port,
|
||||
log_level="warning"))
|
||||
threading.Thread(target=server.run, daemon=True).start()
|
||||
return f"http://{cfg.bind_host}:{cfg.port}"
|
||||
return f"http://{cfg.bind_host}:{cfg.port}", cfg
|
||||
|
||||
|
||||
def install_media(cfg: Config) -> bool:
|
||||
"""Give the selected song a playable folder, so the shot shows the player.
|
||||
|
||||
A flat fill rather than a test pattern: the point is the transport and the
|
||||
lyric tracking underneath it, not the picture. Skipped without ffmpeg.
|
||||
"""
|
||||
if shutil.which("ffmpeg") is None:
|
||||
print("ffmpeg not found; screenshotting without the player")
|
||||
return False
|
||||
artist, title = next((a, t) for t, a, _, _ in SONGS if t == SELECTED)
|
||||
slug = f"{artist} - {title}"
|
||||
out = cfg.output_dir / slug
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-loglevel", "error", "-f", "lavfi",
|
||||
"-i", "color=c=0x1A1430:s=640x360:d=60", "-c:v", "libx264",
|
||||
"-pix_fmt", "yuv420p", "-preset", "ultrafast", "-an",
|
||||
str(out / f"{slug}.mp4")], check=True)
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-loglevel", "error", "-f", "lavfi",
|
||||
"-i", "anullsrc=r=44100:cl=stereo:d=60", "-b:a", "64k",
|
||||
str(out / f"{slug}.mp3")], check=True)
|
||||
(out / f"{slug}.txt").write_text(
|
||||
f"#TITLE:{title}\n#ARTIST:{artist}\n#MP3:{slug}.mp3\n#VIDEO:{slug}.mp4\n"
|
||||
f"#BPM:240\n#GAP:10000\n: 0 4 16 Alas,\nE\n", encoding="utf-8")
|
||||
return True
|
||||
|
||||
|
||||
def seed_jobs(base_url: str) -> None:
|
||||
@@ -187,13 +218,25 @@ def seed_jobs(base_url: str) -> None:
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def capture(base_url: str) -> None:
|
||||
def capture(base_url: str, with_player: bool) -> None:
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 900})
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 1000})
|
||||
page.goto(base_url, wait_until="networkidle")
|
||||
page.get_by_text(SELECTED, exact=True).click()
|
||||
page.wait_for_timeout(300)
|
||||
page.wait_for_timeout(500)
|
||||
if with_player:
|
||||
# paused mid-line, so the shot carries the playhead and the sung /
|
||||
# current word colouring rather than an untouched transport
|
||||
page.evaluate("document.getElementById('player-media').currentTime = 11.6")
|
||||
page.wait_for_timeout(400)
|
||||
# scrolled to the review: the player pins to the top of #main and
|
||||
# the tracked lines sit under it, which is the panel in use
|
||||
page.evaluate("""() => {
|
||||
const main = document.getElementById('main');
|
||||
main.scrollTo(0, document.getElementById('review').offsetTop - 12);
|
||||
}""")
|
||||
page.wait_for_timeout(400)
|
||||
OUT_IMAGE.parent.mkdir(parents=True, exist_ok=True)
|
||||
page.screenshot(path=str(OUT_IMAGE))
|
||||
browser.close()
|
||||
@@ -201,10 +244,11 @@ def capture(base_url: str) -> None:
|
||||
|
||||
def main() -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
base_url = start_server(Path(tmp))
|
||||
base_url, cfg = start_server(Path(tmp))
|
||||
time.sleep(1)
|
||||
seed_jobs(base_url)
|
||||
capture(base_url)
|
||||
with_player = install_media(cfg)
|
||||
capture(base_url, with_player)
|
||||
print(f"wrote {OUT_IMAGE}")
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import os
|
||||
import tomllib
|
||||
from dataclasses import dataclass, fields
|
||||
from dataclasses import dataclass, field, fields
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -18,9 +18,15 @@ class Config:
|
||||
device: str = "cuda"
|
||||
bind_host: str = "127.0.0.1"
|
||||
port: int = 8000
|
||||
# YouTube player clients to try, in order, until one yields a download.
|
||||
# "default" means yt-dlp's own choice; see its extractor-args docs.
|
||||
ytdlp_clients: list[str] = field(default_factory=lambda: ["default", "tv"])
|
||||
ytdlp_cookies: Path | None = None
|
||||
ytdlp_impersonate: str | None = None
|
||||
ytdlp_pot_provider_url: str | None = None
|
||||
|
||||
|
||||
_PATH_FIELDS = {"output_dir", "work_dir"}
|
||||
_PATH_FIELDS = {"output_dir", "work_dir", "ytdlp_cookies"}
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> Config:
|
||||
|
||||
@@ -45,6 +45,12 @@ def song_slug(inp: SongInput) -> str:
|
||||
name = f"{inp.artist} - {inp.title}"
|
||||
return name.replace("/", "").replace("\0", "")
|
||||
|
||||
def read_warnings(work: Path) -> list[str]:
|
||||
notes_file = work / "notes_sec.json"
|
||||
if not notes_file.exists():
|
||||
return []
|
||||
return json.loads(notes_file.read_text()).get("warnings", [])
|
||||
|
||||
def _real_stages() -> list[Stage]:
|
||||
from . import stages
|
||||
return stages.all_stages()
|
||||
@@ -69,8 +75,5 @@ def run_pipeline(inp: SongInput, cfg: Config, log=print,
|
||||
continue
|
||||
log(f"[{stage.name}] running")
|
||||
stage.run(ctx)
|
||||
warnings = []
|
||||
notes_file = work / "notes_sec.json"
|
||||
if notes_file.exists():
|
||||
warnings = json.loads(notes_file.read_text()).get("warnings", [])
|
||||
return PipelineResult(output_dir=cfg.output_dir / song_slug(inp), warnings=warnings)
|
||||
return PipelineResult(output_dir=cfg.output_dir / song_slug(inp),
|
||||
warnings=read_warnings(work))
|
||||
|
||||
@@ -38,6 +38,17 @@ def plan(old: SongInput, new: SongInput) -> str | None:
|
||||
return min((FIELD_STAGE[f] for f in changed), key=STAGE_ORDER.index)
|
||||
|
||||
|
||||
def earliest(*stages: str | None) -> str | None:
|
||||
"""The earliest of the stages given, ignoring None ("nothing forced").
|
||||
|
||||
An operator's explicit choice is combined with plan()'s rather than
|
||||
replacing it: the planner's stage is the latest that is still correct for
|
||||
the edit, so a choice may only ever redo more.
|
||||
"""
|
||||
named = [s for s in stages if s]
|
||||
return min(named, key=STAGE_ORDER.index) if named else None
|
||||
|
||||
|
||||
def relocate_work(old: SongInput, new: SongInput, cfg: Config) -> bool:
|
||||
"""Move the cached work dir when title/artist changed its slug.
|
||||
|
||||
|
||||
@@ -2,14 +2,18 @@ import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from ..config import Config
|
||||
from ..media import ffmpeg
|
||||
from ..pipeline import Ctx, PipelineError, Stage
|
||||
|
||||
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".avif", ".gif"}
|
||||
# yt-dlp's in-progress scratch files, which a failed attempt leaves behind.
|
||||
PARTIAL_EXTENSIONS = {".part", ".ytdl", ".temp"}
|
||||
|
||||
|
||||
def _classify_downloads(dest: Path) -> tuple[Path, Path | None]:
|
||||
files = list(dest.glob("source_video.*"))
|
||||
files = [p for p in dest.glob("source_video.*")
|
||||
if p.suffix.lower() not in PARTIAL_EXTENSIONS]
|
||||
thumbs = [p for p in files if p.suffix.lower() in IMAGE_EXTENSIONS]
|
||||
videos = [p for p in files if p.suffix.lower() not in IMAGE_EXTENSIONS]
|
||||
if not videos:
|
||||
@@ -18,17 +22,67 @@ def _classify_downloads(dest: Path) -> tuple[Path, Path | None]:
|
||||
return video, thumbs[0] if thumbs else None
|
||||
|
||||
|
||||
def download_youtube(url: str, dest: Path) -> tuple[Path, Path | None]:
|
||||
import yt_dlp
|
||||
class _YdlLog:
|
||||
"""Routes yt-dlp's own messages into the job log, minus progress noise."""
|
||||
|
||||
def __init__(self, log):
|
||||
self._log = log
|
||||
|
||||
def debug(self, msg: str) -> None:
|
||||
if not msg.startswith("[debug] "):
|
||||
self._log(msg)
|
||||
|
||||
def warning(self, msg: str) -> None:
|
||||
self._log(msg)
|
||||
|
||||
error = warning
|
||||
|
||||
|
||||
def _ydl_opts(cfg: Config, dest: Path, client: str, log) -> dict:
|
||||
extractor_args: dict[str, dict[str, list[str]]] = {}
|
||||
if client != "default":
|
||||
extractor_args["youtube"] = {"player_client": [client]}
|
||||
if cfg.ytdlp_pot_provider_url:
|
||||
extractor_args["youtubepot-bgutilhttp"] = {
|
||||
"base_url": [cfg.ytdlp_pot_provider_url]}
|
||||
opts = {
|
||||
"outtmpl": str(dest / "source_video.%(ext)s"),
|
||||
"format": "bestvideo*+bestaudio/best",
|
||||
"writethumbnail": True,
|
||||
"noplaylist": True,
|
||||
"noprogress": True,
|
||||
"logger": _YdlLog(log),
|
||||
"retries": 10,
|
||||
"fragment_retries": 10,
|
||||
"extractor_retries": 5,
|
||||
"extractor_args": extractor_args,
|
||||
}
|
||||
with yt_dlp.YoutubeDL(opts) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
return _classify_downloads(dest)
|
||||
if cfg.ytdlp_cookies:
|
||||
opts["cookiefile"] = str(cfg.ytdlp_cookies)
|
||||
if cfg.ytdlp_impersonate:
|
||||
from yt_dlp.networking.impersonate import ImpersonateTarget
|
||||
opts["impersonate"] = ImpersonateTarget.from_str(cfg.ytdlp_impersonate)
|
||||
return opts
|
||||
|
||||
|
||||
def download_youtube(url: str, dest: Path, cfg: Config,
|
||||
log=print) -> tuple[Path, Path | None]:
|
||||
import yt_dlp
|
||||
clients = list(cfg.ytdlp_clients) or ["default"]
|
||||
last: Exception | None = None
|
||||
for client in clients:
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(_ydl_opts(cfg, dest, client, log)) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
return _classify_downloads(dest)
|
||||
except Exception as e:
|
||||
last = e
|
||||
log(f"yt-dlp player_client={client} failed: {e}")
|
||||
raise PipelineError(
|
||||
f"yt-dlp {yt_dlp.version.__version__} could not download {url} after "
|
||||
f"trying player_client={', '.join(clients)}. Last error: {last}. "
|
||||
"YouTube blocks change often: upgrade yt-dlp, then see the yt-dlp "
|
||||
"options in config.toml (cookies, impersonation, PO token provider).")
|
||||
|
||||
|
||||
def _run(ctx: Ctx) -> None:
|
||||
@@ -36,7 +90,8 @@ def _run(ctx: Ctx) -> None:
|
||||
raise PipelineError("provide exactly one of youtube_url / video_path")
|
||||
thumb = None
|
||||
if ctx.inp.youtube_url:
|
||||
video, thumb = download_youtube(ctx.inp.youtube_url, ctx.work)
|
||||
video, thumb = download_youtube(ctx.inp.youtube_url, ctx.work,
|
||||
ctx.cfg, ctx.log)
|
||||
else:
|
||||
video = ctx.work / f"source_video{ctx.inp.video_path.suffix}"
|
||||
shutil.copyfile(ctx.inp.video_path, video)
|
||||
|
||||
@@ -51,11 +51,15 @@ def _run(ctx: Ctx) -> None:
|
||||
meta = json.loads((ctx.work / "meta.json").read_text())
|
||||
src_video = ctx.work / meta["video_file"]
|
||||
fmt, codec = ffprobe_codec(src_video)
|
||||
# The video is background only: the song's audio is #MP3, and the two are
|
||||
# the same recording. A player that does not mute the video plays both at
|
||||
# once, which is heard as an echo, so the audio track is dropped here.
|
||||
if (fmt, codec) in GOOD_VIDEO:
|
||||
shutil.copyfile(src_video, out / f"{slug}{src_video.suffix}")
|
||||
ffmpeg("-i", str(src_video), "-c", "copy", "-an",
|
||||
str(out / f"{slug}{src_video.suffix}"))
|
||||
else:
|
||||
ffmpeg("-i", str(src_video), "-c:v", "libx264", "-crf", "20",
|
||||
"-preset", "medium", "-c:a", "aac", str(out / f"{slug}.mp4"))
|
||||
"-preset", "medium", "-an", str(out / f"{slug}.mp4"))
|
||||
_set_header(txt, "VIDEO", f"{slug}.mp4")
|
||||
cover_src = ctx.inp.cover_path
|
||||
if cover_src is None and meta.get("thumbnail"):
|
||||
|
||||
+353
-23
@@ -111,6 +111,7 @@ body {
|
||||
}
|
||||
|
||||
#sidebar .eyebrow, #jobs .empty-note { padding: 0 18px; }
|
||||
#create-song { display: block; width: calc(100% - 36px); margin: 0 18px 20px; padding: 9px 12px; }
|
||||
|
||||
#jobs { display: flex; flex-direction: column; gap: 1px; margin-top: 8px; }
|
||||
|
||||
@@ -199,6 +200,8 @@ body {
|
||||
.song-form-lyrics textarea { height: 100%; flex: 1 1 auto; }
|
||||
.pair { display: grid; grid-template-columns: 1fr 1fr; gap: 18px; }
|
||||
.field { display: flex; flex-direction: column; gap: 6px; }
|
||||
/* [hidden] is only display:none, which the rule above would otherwise win */
|
||||
.field[hidden], .keep-note[hidden] { display: none; }
|
||||
.field > span {
|
||||
font-family: var(--font-display);
|
||||
font-weight: 600;
|
||||
@@ -307,6 +310,7 @@ button { cursor: pointer; }
|
||||
transition: background 120ms ease;
|
||||
}
|
||||
.primary:hover { background: #BE7BF9; }
|
||||
.primary:disabled { background: var(--control); color: var(--text-muted); cursor: not-allowed; }
|
||||
|
||||
#form-error { margin: 12px 0 0; min-height: 1.2em; font-size: 0.85rem; color: var(--fail); }
|
||||
|
||||
@@ -319,8 +323,27 @@ button { cursor: pointer; }
|
||||
}
|
||||
#review h3 span { color: var(--text-muted); font-weight: 400; }
|
||||
|
||||
/* sticks to the top of #main so the video stays in frame as the lyrics scroll */
|
||||
#player { position: sticky; top: 0; z-index: 2; margin: 0 0 14px; padding: 6px 0 10px; background: var(--bg); }
|
||||
#player-video {
|
||||
display: block;
|
||||
width: 100%;
|
||||
max-height: 30vh;
|
||||
background: #000;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 2px;
|
||||
}
|
||||
#player-video[hidden] { display: none; }
|
||||
/* the native transport renders light otherwise, against everything around it */
|
||||
#player-media { display: block; width: 100%; margin-top: 8px; color-scheme: dark; }
|
||||
|
||||
#lyrics-review { display: flex; flex-direction: column; gap: 10px; }
|
||||
.review-line { padding: 8px 12px; background: var(--panel); border: 1px solid var(--border); }
|
||||
.review-line-active { background: var(--panel-raised); border-color: var(--aqua); }
|
||||
.review-word-sung { color: var(--text-muted); }
|
||||
.review-word-active { color: var(--aqua); font-weight: 600; }
|
||||
.review-playhead { position: absolute; top: 0; bottom: 0; width: 2px; margin-left: -1px; background: var(--aqua); }
|
||||
.review-seek { cursor: pointer; }
|
||||
.review-line-text { display: block; font-size: 0.9rem; }
|
||||
.review-line-dropped { opacity: 0.55; }
|
||||
.review-line-dropped .review-line-text { text-decoration: line-through; }
|
||||
@@ -354,7 +377,7 @@ button { cursor: pointer; }
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
.output-path code { flex: 1; overflow-wrap: anywhere; font-family: var(--font-mono); font-size: 0.82rem; }
|
||||
#copy-path, .action, #cancel-rebuild {
|
||||
#copy-path, .action {
|
||||
flex: none;
|
||||
padding: 5px 12px;
|
||||
font-family: var(--font-display);
|
||||
@@ -367,10 +390,11 @@ button { cursor: pointer; }
|
||||
border: 1px solid var(--control);
|
||||
border-radius: 2px;
|
||||
}
|
||||
#copy-path:hover, .action:hover, #cancel-rebuild:hover { border-color: var(--text-muted); }
|
||||
#copy-path:hover, .action:hover { border-color: var(--text-muted); }
|
||||
|
||||
.form-actions { display: flex; align-items: center; gap: 14px; }
|
||||
.keep-note { margin: -8px 0 0; font-size: 0.78rem; color: var(--aqua); }
|
||||
#form-lock { margin: 0; font-size: 0.78rem; color: var(--text-muted); }
|
||||
#review-actions { display: flex; gap: 10px; margin-top: 14px; }
|
||||
#review-lineage { margin: -10px 0 14px; }
|
||||
.lineage {
|
||||
@@ -493,6 +517,7 @@ button { cursor: pointer; }
|
||||
<div class="kana" aria-hidden="true">カラオケ</div>
|
||||
</div>
|
||||
<div>
|
||||
<button type="button" id="create-song" class="primary">CREATE SONG</button>
|
||||
<h2 class="eyebrow">QUEUE</h2>
|
||||
<div id="jobs"></div>
|
||||
</div>
|
||||
@@ -511,6 +536,10 @@ button { cursor: pointer; }
|
||||
<label class="field"><span>YouTube URL</span><input name="youtube_url"></label>
|
||||
<label class="field"><span>Video file</span><input type="file" name="video"></label>
|
||||
<p class="keep-note" data-keep="video" hidden></p>
|
||||
<label class="field" id="restart-field" hidden>
|
||||
<span>Restart from</span>
|
||||
<select name="force_from"></select>
|
||||
</label>
|
||||
<details>
|
||||
<summary>Advanced</summary>
|
||||
<div class="stack">
|
||||
@@ -531,7 +560,7 @@ button { cursor: pointer; }
|
||||
</div>
|
||||
<div class="form-actions">
|
||||
<button type="submit" class="primary">CREATE SONG</button>
|
||||
<button type="button" id="cancel-rebuild" hidden>CANCEL</button>
|
||||
<p id="form-lock" hidden></p>
|
||||
</div>
|
||||
</form>
|
||||
<p id="form-error" role="alert"></p>
|
||||
@@ -542,6 +571,10 @@ button { cursor: pointer; }
|
||||
<h3 id="review-title"></h3>
|
||||
<p id="review-lineage" hidden></p>
|
||||
<p id="review-error" class="review-error" hidden></p>
|
||||
<div id="player" hidden>
|
||||
<video id="player-video" muted playsinline preload="metadata"></video>
|
||||
<audio id="player-media" controls preload="metadata"></audio>
|
||||
</div>
|
||||
<div id="lyrics-review"><p class="empty-note">no songs queued yet — nothing to review</p></div>
|
||||
<div id="output-path"></div>
|
||||
<div id="review-actions"></div>
|
||||
@@ -572,10 +605,15 @@ const nowTitle = document.getElementById("now-title");
|
||||
const stageTrack = document.getElementById("stage-track");
|
||||
const reviewActions = document.getElementById("review-actions");
|
||||
const reviewLineage = document.getElementById("review-lineage");
|
||||
const cancelRebuild = document.getElementById("cancel-rebuild");
|
||||
const createSong = document.getElementById("create-song");
|
||||
const formLock = document.getElementById("form-lock");
|
||||
const formHeading = document.querySelector("#new-song h2");
|
||||
const submitButton = form.querySelector("button[type=submit]");
|
||||
let rebuildOf = null;
|
||||
let formDirty = false;
|
||||
// set by CREATE SONG: the page is a blank sheet on purpose, so the poll must
|
||||
// not quietly select the newest song back into it
|
||||
let composing = false;
|
||||
let JOBS = [];
|
||||
let selectedId = null;
|
||||
let STAGES = [];
|
||||
@@ -610,10 +648,16 @@ form.addEventListener("submit", async (e) => {
|
||||
if (!resp.ok) {
|
||||
const body = await resp.json().catch(() => ({}));
|
||||
formError.textContent = body.detail || `error: ${resp.status}`;
|
||||
return; // before the reset below, so a rejected submission is never retyped
|
||||
return;
|
||||
}
|
||||
selectedId = (await resp.json()).id;
|
||||
exitRebuild();
|
||||
composing = false;
|
||||
// the fields stay as submitted: the song just sent is the obvious starting
|
||||
// point for the next one, and nothing has to be retyped to correct it. The
|
||||
// queue's CREATE SONG button is what empties the form.
|
||||
rebuildOf = null;
|
||||
formDirty = false;
|
||||
setFormMode(null, null);
|
||||
refreshJobs();
|
||||
});
|
||||
|
||||
@@ -638,7 +682,25 @@ for (const input of form.querySelectorAll('input[type="file"]')) {
|
||||
});
|
||||
}
|
||||
|
||||
cancelRebuild.addEventListener("click", exitRebuild);
|
||||
// selecting another queue entry reloads the form, so track whether there is
|
||||
// anything of the operator's in there worth warning about first
|
||||
form.addEventListener("input", () => { formDirty = true; });
|
||||
|
||||
// Always available, and the form is usually holding a selected song by now, so
|
||||
// this clears the whole page down to a blank sheet — fields, review, player and
|
||||
// queue selection — and brings the form back into view from wherever playback
|
||||
// scrolled the panel to. Leaving another song's review under an empty form
|
||||
// reads as though the two belong together.
|
||||
createSong.addEventListener("click", () => {
|
||||
if (formDirty && !confirm("Discard your unsaved edits to this form?")) return;
|
||||
composing = true;
|
||||
selectedId = null;
|
||||
for (const card of jobsDiv.querySelectorAll(".job")) card.removeAttribute("aria-current");
|
||||
exitRebuild();
|
||||
clearDetail("nothing selected");
|
||||
document.getElementById("new-song").scrollIntoView({ block: "start", behavior: "smooth" });
|
||||
form.elements.title.focus();
|
||||
});
|
||||
|
||||
function jobCard(job) {
|
||||
const card = document.createElement("button");
|
||||
@@ -713,6 +775,11 @@ function renderJobs(jobs) {
|
||||
async function loadStages() {
|
||||
const resp = await fetch("/api/stages");
|
||||
STAGES = (await resp.json()).stages;
|
||||
// a rebuild redoes only what the edit invalidated, which cannot see a change
|
||||
// in songtrace itself — this is how you redo a stage anyway
|
||||
const select = form.elements.force_from;
|
||||
select.replaceChildren(new Option("only what changed", ""),
|
||||
...STAGES.map(s => new Option(s, s)));
|
||||
}
|
||||
|
||||
function renderStageTrack(job) {
|
||||
@@ -772,19 +839,28 @@ async function refreshJobs() {
|
||||
const resp = await fetch("/api/jobs");
|
||||
const jobs = await resp.json();
|
||||
JOBS = jobs;
|
||||
if (selectedId === null && jobs.length) selectedId = jobs[0].id;
|
||||
// deliberately not via selectJob: the auto-selection on load fills the review
|
||||
// panel but leaves the form blank, so landing on the page is still the way to
|
||||
// start a new song. Clicking a queue entry is what loads its inputs.
|
||||
if (!composing && selectedId === null && jobs.length) selectedId = jobs[0].id;
|
||||
renderJobs(jobs);
|
||||
renderNowBar(jobs);
|
||||
if (selectedId) refreshDetail();
|
||||
else clearDetail(composing ? "nothing selected" : undefined);
|
||||
schedulePoll(jobs);
|
||||
}
|
||||
|
||||
async function selectJob(id) {
|
||||
// the form already holds this job; reloading it would drop edits in progress
|
||||
const reload = id !== rebuildOf;
|
||||
if (reload && formDirty && !confirm("Discard your unsaved edits to this form?")) return;
|
||||
composing = false;
|
||||
selectedId = id;
|
||||
for (const card of jobsDiv.querySelectorAll(".job")) {
|
||||
if (card.dataset.id === id) card.setAttribute("aria-current", "true");
|
||||
else card.removeAttribute("aria-current");
|
||||
}
|
||||
if (reload) await loadInputs(id, JOBS.find(j => j.id === id)?.status);
|
||||
refreshDetail();
|
||||
}
|
||||
|
||||
@@ -856,6 +932,10 @@ function lineTimeline(words, notes, minPitch, span) {
|
||||
}
|
||||
box.append(wordsRow);
|
||||
if (notes.length) box.append(pianoRoll(notes, minPitch, span, t0, t1));
|
||||
const playhead = document.createElement("div");
|
||||
playhead.className = "review-playhead";
|
||||
playhead.hidden = true;
|
||||
box.append(playhead);
|
||||
return box;
|
||||
}
|
||||
|
||||
@@ -889,6 +969,21 @@ function lineNode(line, minPitch, span) {
|
||||
return row;
|
||||
}
|
||||
|
||||
// What playback needs from a rendered line: the time range it was laid out
|
||||
// over, the word labels along it, and the bar that sweeps across it.
|
||||
function lineEntry(row, line) {
|
||||
const laid = line.words.length ? line.words : line.notes;
|
||||
if (!laid.length || line.dropped) return null;
|
||||
const labels = [...row.querySelectorAll(".review-word")];
|
||||
return {
|
||||
row,
|
||||
t0: laid[0].start,
|
||||
t1: Math.max(...laid.map(x => x.end)),
|
||||
words: line.words.map((w, i) => ({ el: labels[i], start: w.start, end: w.end })),
|
||||
playhead: row.querySelector(".review-playhead"),
|
||||
};
|
||||
}
|
||||
|
||||
function buildLyricsReview(data) {
|
||||
const allNotes = data.lines.flatMap(l => l.notes);
|
||||
const pitches = allNotes.map(n => n.pitch);
|
||||
@@ -897,6 +992,7 @@ function buildLyricsReview(data) {
|
||||
const span = Math.max(1, maxPitch - minPitch);
|
||||
const spans = [...data.missing_spans].sort((a, b) => a.start - b.start);
|
||||
const nodes = [];
|
||||
const entries = [];
|
||||
let spanIdx = 0;
|
||||
for (const line of data.lines) {
|
||||
const lineStart = line.notes.length ? Math.min(...line.notes.map(n => n.start)) : null;
|
||||
@@ -904,24 +1000,172 @@ function buildLyricsReview(data) {
|
||||
nodes.push(missingSpanNode(spans[spanIdx]));
|
||||
spanIdx++;
|
||||
}
|
||||
nodes.push(lineNode(line, minPitch, span));
|
||||
const row = lineNode(line, minPitch, span);
|
||||
const entry = lineEntry(row, line);
|
||||
if (entry) entries.push(entry);
|
||||
nodes.push(row);
|
||||
}
|
||||
while (spanIdx < spans.length) {
|
||||
nodes.push(missingSpanNode(spans[spanIdx]));
|
||||
spanIdx++;
|
||||
}
|
||||
return nodes;
|
||||
return { nodes, entries };
|
||||
}
|
||||
|
||||
// The song folder's own media, played against the same seconds the review was
|
||||
// built from, so the highlighted line is the one you are hearing.
|
||||
//
|
||||
// Modelled on the game: #MP3 is the audio and the clock, the video is muted
|
||||
// background slaved to it. Playing the video's own audio alongside the mp3 is
|
||||
// the same recording twice, a few frames apart, which is heard as an echo.
|
||||
const player = document.getElementById("player");
|
||||
const media = document.getElementById("player-media");
|
||||
const video = document.getElementById("player-video");
|
||||
const DRIFT = 0.25; // seconds before the picture is worth re-seeking
|
||||
|
||||
function slaveVideo(force) {
|
||||
if (video.hidden || !video.getAttribute("src")) return;
|
||||
if (force || Math.abs(video.currentTime - media.currentTime) > DRIFT) {
|
||||
video.currentTime = media.currentTime;
|
||||
}
|
||||
}
|
||||
|
||||
for (const event of ["seeked", "ratechange"]) {
|
||||
media.addEventListener(event, () => {
|
||||
video.playbackRate = media.playbackRate;
|
||||
slaveVideo(true);
|
||||
});
|
||||
}
|
||||
media.addEventListener("pause", () => video.pause());
|
||||
let timeline = [];
|
||||
let activeEntry = null;
|
||||
let syncFrame = null;
|
||||
|
||||
// the first line that has not finished yet: the one being sung, or the one
|
||||
// coming up while the track is between lines
|
||||
function entryAt(t) {
|
||||
return timeline.find(e => t <= e.t1) ?? timeline[timeline.length - 1] ?? null;
|
||||
}
|
||||
|
||||
function paintEntry(entry, t) {
|
||||
for (const w of entry.words) {
|
||||
w.el.classList.toggle("review-word-sung", t >= w.end);
|
||||
w.el.classList.toggle("review-word-active", t >= w.start && t < w.end);
|
||||
}
|
||||
if (!entry.playhead) return;
|
||||
const inside = t >= entry.t0 && t <= entry.t1;
|
||||
entry.playhead.hidden = !inside;
|
||||
if (inside) {
|
||||
const frac = (t - entry.t0) / Math.max(entry.t1 - entry.t0, 0.001);
|
||||
entry.playhead.style.left = `${frac * 100}%`;
|
||||
}
|
||||
}
|
||||
|
||||
function clearEntry(entry) {
|
||||
entry.row.classList.remove("review-line-active");
|
||||
if (entry.playhead) entry.playhead.hidden = true;
|
||||
for (const w of entry.words) {
|
||||
w.el.classList.remove("review-word-sung", "review-word-active");
|
||||
}
|
||||
}
|
||||
|
||||
function syncPlayback() {
|
||||
const t = media.currentTime;
|
||||
const entry = entryAt(t);
|
||||
if (entry !== activeEntry) {
|
||||
if (activeEntry) clearEntry(activeEntry);
|
||||
activeEntry = entry;
|
||||
if (entry) {
|
||||
entry.row.classList.add("review-line-active");
|
||||
focusActive();
|
||||
}
|
||||
}
|
||||
if (entry) paintEntry(entry, t);
|
||||
}
|
||||
|
||||
// centred so there is context either side, like a scrolling lyric sheet; only
|
||||
// while running, so it does not fight a reader who has scrolled away to look
|
||||
// at something else
|
||||
function focusActive() {
|
||||
if (!activeEntry || media.paused) return;
|
||||
activeEntry.row.scrollIntoView({ block: "center", behavior: "smooth" });
|
||||
}
|
||||
|
||||
function pumpPlayback() {
|
||||
syncPlayback();
|
||||
slaveVideo(false);
|
||||
syncFrame = media.paused ? null : requestAnimationFrame(pumpPlayback);
|
||||
}
|
||||
|
||||
// timeupdate fires about four times a second, too coarse for a playhead
|
||||
media.addEventListener("play", () => {
|
||||
if (syncFrame === null) syncFrame = requestAnimationFrame(pumpPlayback);
|
||||
if (!video.hidden && video.getAttribute("src")) video.play().catch(() => {});
|
||||
focusActive(); // the line may already be active from a seek while paused
|
||||
});
|
||||
for (const event of ["pause", "seeked", "loadedmetadata"]) {
|
||||
media.addEventListener(event, syncPlayback);
|
||||
}
|
||||
|
||||
function bindTimeline(entries) {
|
||||
if (activeEntry) clearEntry(activeEntry);
|
||||
activeEntry = null;
|
||||
timeline = entries;
|
||||
for (const entry of entries) {
|
||||
entry.row.classList.add("review-seek");
|
||||
entry.row.title = "click to play from here";
|
||||
entry.row.addEventListener("click", (e) => {
|
||||
const word = entry.words.find(w => w.el === e.target);
|
||||
media.currentTime = word ? word.start : entry.t0;
|
||||
if (media.paused) syncPlayback();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Returns true when the folder gave us something to play. #MP3 is required —
|
||||
// it is the audio the game uses, so it is the one the review has to track.
|
||||
function setPlayerSource(job, available) {
|
||||
const src = available?.audio ? `/api/jobs/${job.id}/media/audio` : "";
|
||||
player.hidden = !src;
|
||||
if (media.dataset.src === src) return !!src;
|
||||
media.dataset.src = src;
|
||||
if (src) media.src = src;
|
||||
else { media.removeAttribute("src"); media.load(); }
|
||||
|
||||
video.hidden = !(src && available.video);
|
||||
video.poster = available?.cover ? `/api/jobs/${job.id}/media/cover` : "";
|
||||
if (!video.hidden) video.src = `/api/jobs/${job.id}/media/video`;
|
||||
else { video.removeAttribute("src"); video.load(); }
|
||||
return !!src;
|
||||
}
|
||||
|
||||
function resetPlayer() {
|
||||
// the 3s poll reaches here for every running job; reloading a cleared
|
||||
// element each time would be pure churn
|
||||
if (player.hidden && !timeline.length) return;
|
||||
bindTimeline([]);
|
||||
player.hidden = true;
|
||||
media.pause();
|
||||
video.pause();
|
||||
media.dataset.src = "";
|
||||
for (const el of [media, video]) {
|
||||
el.removeAttribute("src");
|
||||
el.removeAttribute("poster");
|
||||
el.load();
|
||||
}
|
||||
}
|
||||
|
||||
async function renderLyricsReview(job) {
|
||||
const box = document.getElementById("lyrics-review");
|
||||
if (job.status === "failed") {
|
||||
delete box.dataset.jobId;
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("job failed — nothing to review"));
|
||||
return;
|
||||
}
|
||||
if (job.status !== "done") {
|
||||
delete box.dataset.jobId;
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("waiting for this song to finish…"));
|
||||
return;
|
||||
}
|
||||
@@ -929,6 +1173,7 @@ async function renderLyricsReview(job) {
|
||||
const resp = await fetch(`/api/jobs/${job.id}/review`);
|
||||
if (job.id !== selectedId) return;
|
||||
if (!resp.ok) {
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("no review data available"));
|
||||
return;
|
||||
}
|
||||
@@ -936,10 +1181,17 @@ async function renderLyricsReview(job) {
|
||||
if (job.id !== selectedId) return;
|
||||
box.dataset.jobId = job.id;
|
||||
if (!data.lines.length) {
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("no lyrics to review"));
|
||||
return;
|
||||
}
|
||||
box.replaceChildren(...buildLyricsReview(data));
|
||||
const { nodes, entries } = buildLyricsReview(data);
|
||||
box.replaceChildren(...nodes);
|
||||
// with nothing to play, a highlighted line and a click-to-seek cursor would
|
||||
// both be promising something the song folder cannot deliver
|
||||
const playable = setPlayerSource(job, data.media);
|
||||
bindTimeline(playable ? entries : []);
|
||||
if (playable) syncPlayback();
|
||||
}
|
||||
|
||||
async function copyPath(path, button) {
|
||||
@@ -999,25 +1251,63 @@ function renderOutputPath(job) {
|
||||
outputPathDiv.replaceChildren(wrap);
|
||||
}
|
||||
|
||||
function setFormMode(inp) {
|
||||
function clearDetail(why = "no songs queued yet") {
|
||||
reviewTitle.replaceChildren();
|
||||
reviewError.hidden = true;
|
||||
reviewError.textContent = "";
|
||||
resetPlayer();
|
||||
// the review box, not an element that no longer exists: getting this wrong
|
||||
// threw partway through and left the rest of the panel showing the old song
|
||||
const box = document.getElementById("lyrics-review");
|
||||
delete box.dataset.jobId;
|
||||
box.replaceChildren(emptyNote(`${why} — nothing to review`));
|
||||
outputPathDiv.replaceChildren();
|
||||
reviewActions.replaceChildren();
|
||||
delete reviewActions.dataset.jobId;
|
||||
reviewLineage.replaceChildren();
|
||||
reviewLineage.hidden = true;
|
||||
delete reviewLineage.dataset.jobId;
|
||||
detail.textContent = `${why} — nothing to log`;
|
||||
}
|
||||
|
||||
// a rebuild is refused with 409 until the job settles, so the form says so
|
||||
// rather than letting the submission fail
|
||||
function setSubmitLock(status) {
|
||||
const locked = rebuildOf !== null && (status === "queued" || status === "running");
|
||||
submitButton.disabled = locked;
|
||||
formLock.textContent = locked ? "editable once this run finishes" : "";
|
||||
formLock.hidden = !locked;
|
||||
}
|
||||
|
||||
function setFormMode(inp, status) {
|
||||
formHeading.textContent = rebuildOf ? "REBUILD" : "NEW SONG";
|
||||
submitButton.textContent = rebuildOf ? "REBUILD SONG" : "CREATE SONG";
|
||||
cancelRebuild.hidden = !rebuildOf;
|
||||
setSubmitLock(status);
|
||||
// only a rebuild has stages to restart from; a new song always runs them all
|
||||
document.getElementById("restart-field").hidden = !rebuildOf;
|
||||
if (!rebuildOf) form.elements.force_from.value = "";
|
||||
for (const [field, key] of [["video", "video_filename"], ["cover", "cover_filename"]]) {
|
||||
const note = form.querySelector(`[data-keep="${field}"]`);
|
||||
const kept = rebuildOf && inp ? inp[key] : null;
|
||||
note.textContent = kept ? `keeping ${kept} — attach a file to replace it` : "";
|
||||
note.hidden = !kept;
|
||||
}
|
||||
// the advanced fields carry into a rebuild whether or not the section is open,
|
||||
// so anything actually set must not sit hidden behind a closed summary
|
||||
form.querySelector("details").open = !!inp?.cover_filename ||
|
||||
["genre", "year", "language", "bpm"].some(n => form.elements[n].value);
|
||||
}
|
||||
|
||||
function exitRebuild() {
|
||||
rebuildOf = null;
|
||||
form.reset();
|
||||
setFormMode(null);
|
||||
formDirty = false;
|
||||
setFormMode(null, null);
|
||||
}
|
||||
|
||||
async function startRebuild(id) {
|
||||
// selecting a queue entry brings its inputs back up, so a song can be adjusted
|
||||
// and rebuilt without being retyped
|
||||
async function loadInputs(id, status) {
|
||||
const resp = await fetch(`/api/jobs/${id}/input`);
|
||||
if (!resp.ok) return;
|
||||
const inp = await resp.json();
|
||||
@@ -1028,9 +1318,9 @@ async function startRebuild(id) {
|
||||
if (el && value !== null) el.value = value;
|
||||
}
|
||||
rebuildOf = id;
|
||||
setFormMode(inp);
|
||||
formDirty = false;
|
||||
setFormMode(inp, status);
|
||||
formError.textContent = "";
|
||||
form.elements.title.focus();
|
||||
}
|
||||
|
||||
async function rerunJob(id) {
|
||||
@@ -1045,21 +1335,60 @@ async function rerunJob(id) {
|
||||
refreshJobs();
|
||||
}
|
||||
|
||||
async function rebuildFullJob(id) {
|
||||
formError.textContent = "";
|
||||
const resp = await fetch(`/api/jobs/${id}/rebuild-full`, { method: "POST" });
|
||||
if (!resp.ok) {
|
||||
const body = await resp.json().catch(() => ({}));
|
||||
formError.textContent = body.detail || `error: ${resp.status}`;
|
||||
return;
|
||||
}
|
||||
selectedId = (await resp.json()).id;
|
||||
refreshJobs();
|
||||
}
|
||||
|
||||
async function deleteJob(id) {
|
||||
if (!confirm("Delete this song? This removes all its files and cannot be undone.")) return;
|
||||
formError.textContent = "";
|
||||
const resp = await fetch(`/api/jobs/${id}`, { method: "DELETE" });
|
||||
if (!resp.ok) {
|
||||
const body = await resp.json().catch(() => ({}));
|
||||
formError.textContent = body.detail || `error: ${resp.status}`;
|
||||
return;
|
||||
}
|
||||
if (selectedId === id) selectedId = null;
|
||||
// the form is holding a job that no longer exists; a rebuild would 404
|
||||
if (rebuildOf === id) exitRebuild();
|
||||
refreshJobs();
|
||||
}
|
||||
|
||||
const ACTION_LABELS = {
|
||||
rerun: "RERUN",
|
||||
"rebuild-full": "REBUILD FROM SCRATCH",
|
||||
delete: "DELETE",
|
||||
};
|
||||
|
||||
const ACTION_HANDLERS = {
|
||||
rerun: rerunJob,
|
||||
"rebuild-full": rebuildFullJob,
|
||||
delete: deleteJob,
|
||||
};
|
||||
|
||||
function actionButton(action, jobId) {
|
||||
const button = document.createElement("button");
|
||||
button.type = "button";
|
||||
button.className = "action";
|
||||
button.dataset.action = action;
|
||||
button.textContent = action === "rerun" ? "RERUN" : "EDIT & REBUILD";
|
||||
button.addEventListener("click",
|
||||
() => action === "rerun" ? rerunJob(jobId) : startRebuild(jobId));
|
||||
button.textContent = ACTION_LABELS[action];
|
||||
button.addEventListener("click", () => ACTION_HANDLERS[action](jobId));
|
||||
return button;
|
||||
}
|
||||
|
||||
function renderActions(job) {
|
||||
// no controls mid-run: both endpoints answer 409 until the job settles
|
||||
const wanted = job.status === "failed" ? ["rerun", "edit"]
|
||||
: job.status === "done" ? ["edit"] : [];
|
||||
// no controls mid-run: every endpoint answers 409 until the job settles
|
||||
const doneActions = ["rebuild-full", "delete"];
|
||||
const wanted = job.status === "failed" ? ["rerun", ...doneActions]
|
||||
: job.status === "done" ? doneActions : [];
|
||||
const shown = [...reviewActions.children].map(b => b.dataset.action);
|
||||
// rebuilding an unchanged row would detach a focused action on every poll
|
||||
if (reviewActions.dataset.jobId === job.id && shown.join() === wanted.join()) return;
|
||||
@@ -1094,6 +1423,7 @@ async function refreshDetail() {
|
||||
reviewTitle.replaceChildren(`${job.title} `, artist);
|
||||
reviewError.hidden = !job.error;
|
||||
reviewError.textContent = job.error ?? "";
|
||||
if (rebuildOf === job.id) setSubmitLock(job.status);
|
||||
renderLyricsReview(job);
|
||||
renderOutputPath(job);
|
||||
renderLineage(job);
|
||||
|
||||
+126
-12
@@ -6,7 +6,7 @@ import shutil
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
|
||||
@@ -17,11 +17,14 @@ from fastapi.staticfiles import StaticFiles
|
||||
from . import rebuild as rebuild_policy
|
||||
from .config import Config
|
||||
from .lyrics import normalize_lyrics
|
||||
from .pipeline import (STAGE_ORDER, PipelineError, SongInput, run_pipeline,
|
||||
song_slug)
|
||||
from .pipeline import (STAGE_ORDER, PipelineError, SongInput, read_warnings,
|
||||
run_pipeline, song_slug)
|
||||
from .ultrastar import get_header, parse_ultrastar
|
||||
|
||||
STATIC = Path(__file__).parent / "static"
|
||||
STAGE_LINE = re.compile(r"^\[(\w+)\] (?:running|cached, skipping)")
|
||||
# the song folder names its own media, exactly as the game reads it
|
||||
MEDIA_HEADERS = {"audio": "MP3", "video": "VIDEO", "cover": "COVER"}
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -37,6 +40,7 @@ class Job:
|
||||
error: str | None = None
|
||||
inp: SongInput | None = None
|
||||
rebuilt_from: str | None = None
|
||||
force_stage: str | None = None
|
||||
created_at: float = field(default_factory=time.time)
|
||||
started_at: float | None = None
|
||||
finished_at: float | None = None
|
||||
@@ -93,6 +97,35 @@ def _cleanup_uploads(cfg: Config, job_id: str, video_path: Path | None) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _discover_jobs(cfg: Config) -> list[Job]:
|
||||
discovered = []
|
||||
if not cfg.work_dir.exists():
|
||||
return discovered
|
||||
for sidecar in cfg.work_dir.glob("*/input.json"):
|
||||
try:
|
||||
data = json.loads(sidecar.read_text())
|
||||
for key in ("video_path", "cover_path"):
|
||||
if data.get(key) is not None:
|
||||
data[key] = Path(data[key])
|
||||
inp = SongInput(**data)
|
||||
slug = song_slug(inp)
|
||||
out = cfg.output_dir / slug
|
||||
done = out.exists() and any(out.iterdir())
|
||||
mtime = sidecar.stat().st_mtime
|
||||
warnings = read_warnings(sidecar.parent) if done else []
|
||||
except (OSError, ValueError, TypeError):
|
||||
continue
|
||||
discovered.append(Job(
|
||||
id=uuid.uuid4().hex, title=inp.title, artist=inp.artist, inp=inp,
|
||||
status="done" if done else "failed",
|
||||
output_dir=str(out) if done else None,
|
||||
warnings=warnings,
|
||||
error=None if done else "interrupted before completion",
|
||||
created_at=mtime, finished_at=mtime))
|
||||
discovered.sort(key=lambda j: j.created_at)
|
||||
return discovered
|
||||
|
||||
|
||||
def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
app = FastAPI()
|
||||
app.mount("/static", StaticFiles(directory=STATIC), name="static")
|
||||
@@ -101,17 +134,24 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
lock = threading.Lock()
|
||||
todo: queue.Queue[str] = queue.Queue()
|
||||
|
||||
for job in _discover_jobs(cfg):
|
||||
jobs[job.id] = job
|
||||
order.append(job.id)
|
||||
|
||||
def _prepare_rebuild(job: Job, log) -> str | None:
|
||||
"""Relocate cached work and report the stage this rebuild restarts from."""
|
||||
if not job.rebuilt_from:
|
||||
return None
|
||||
return job.force_stage
|
||||
with lock:
|
||||
parent = jobs.get(job.rebuilt_from)
|
||||
if parent is None:
|
||||
return None
|
||||
force_from = rebuild_policy.plan(parent.inp, job.inp)
|
||||
return job.force_stage
|
||||
planned = rebuild_policy.plan(parent.inp, job.inp)
|
||||
changed = rebuild_policy.changed_fields(parent.inp, job.inp)
|
||||
rebuild_policy.relocate_work(parent.inp, job.inp, cfg)
|
||||
# a stage the operator asked for can only widen what plan() decided,
|
||||
# which is how "redo the packaging" reaches work nothing invalidated
|
||||
force_from = rebuild_policy.earliest(planned, job.force_stage)
|
||||
log(f"[rebuild] restarting from {force_from or 'the first incomplete stage'}"
|
||||
f" ({', '.join(changed) if changed else 'nothing'} changed)")
|
||||
return force_from
|
||||
@@ -130,6 +170,15 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
shutil.rmtree(old, ignore_errors=True)
|
||||
log(f"[rebuild] removed superseded {old.name}")
|
||||
|
||||
def _write_sidecar(job: Job) -> None:
|
||||
work = cfg.work_dir / song_slug(job.inp)
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
data = asdict(job.inp)
|
||||
for key in ("video_path", "cover_path"):
|
||||
if data[key] is not None:
|
||||
data[key] = str(data[key])
|
||||
(work / "input.json").write_text(json.dumps(data))
|
||||
|
||||
def worker():
|
||||
while True:
|
||||
job_id = todo.get()
|
||||
@@ -146,6 +195,7 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
|
||||
try:
|
||||
force_from = _prepare_rebuild(job, log)
|
||||
_write_sidecar(job)
|
||||
result = runner(job.inp, cfg, log=log, force_from=force_from)
|
||||
with lock:
|
||||
job.output_dir = str(result.output_dir)
|
||||
@@ -166,10 +216,10 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
|
||||
threading.Thread(target=worker, daemon=True).start()
|
||||
|
||||
def _enqueue(job_id: str, inp: SongInput,
|
||||
rebuilt_from: str | None = None) -> dict:
|
||||
def _enqueue(job_id: str, inp: SongInput, rebuilt_from: str | None = None,
|
||||
force_stage: str | None = None) -> dict:
|
||||
job = Job(id=job_id, title=inp.title, artist=inp.artist, inp=inp,
|
||||
rebuilt_from=rebuilt_from)
|
||||
rebuilt_from=rebuilt_from, force_stage=force_stage)
|
||||
with lock:
|
||||
jobs[job_id] = job
|
||||
order.append(job_id)
|
||||
@@ -198,6 +248,7 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
cover: UploadFile | None = File(None)):
|
||||
youtube_url, genre, language = (youtube_url or None, genre or None,
|
||||
language or None)
|
||||
lyrics = lyrics.replace("\r\n", "\n")
|
||||
has_video = bool(video and video.filename)
|
||||
if bool(youtube_url) == has_video:
|
||||
raise HTTPException(400, "provide exactly one of youtube_url / video")
|
||||
@@ -218,6 +269,12 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
parent = _finished_job(job_id)
|
||||
return _enqueue(uuid.uuid4().hex, parent.inp, rebuilt_from=job_id)
|
||||
|
||||
@app.post("/api/jobs/{job_id}/rebuild-full", status_code=201)
|
||||
def rebuild_full(job_id: str):
|
||||
parent = _finished_job(job_id)
|
||||
return _enqueue(uuid.uuid4().hex, parent.inp, rebuilt_from=job_id,
|
||||
force_stage=STAGE_ORDER[0])
|
||||
|
||||
def _require_source(parent_inp: SongInput, inp: SongInput,
|
||||
force_from: str | None) -> None:
|
||||
"""Refuse a rebuild that will rerun acquire against an upload we no longer have."""
|
||||
@@ -237,13 +294,20 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
lyrics: str = Form(...), youtube_url: str | None = Form(None),
|
||||
genre: str | None = Form(None), year: int | None = Form(None),
|
||||
language: str | None = Form(None), bpm: float | None = Form(None),
|
||||
force_from: str | None = Form(None),
|
||||
video: UploadFile | None = File(None),
|
||||
cover: UploadFile | None = File(None)):
|
||||
parent = _finished_job(job_id)
|
||||
force_from = force_from or None
|
||||
if force_from is not None and force_from not in STAGE_ORDER:
|
||||
raise HTTPException(400, f"unknown stage {force_from!r}")
|
||||
# a cleared field arrives as "" from some clients; "" and None must not
|
||||
# look like different inputs to plan()
|
||||
youtube_url, genre, language = (youtube_url or None, genre or None,
|
||||
language or None)
|
||||
# a browser submits a textarea with CRLF; without this every rebuild
|
||||
# from the page looks like a lyrics edit and redoes the alignment
|
||||
lyrics = lyrics.replace("\r\n", "\n")
|
||||
has_video = bool(video and video.filename)
|
||||
if youtube_url and has_video:
|
||||
raise HTTPException(400, "provide exactly one of youtube_url / video")
|
||||
@@ -261,8 +325,9 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
youtube_url=youtube_url, video_path=video_path,
|
||||
genre=genre, year=year, language=language,
|
||||
cover_path=cover_path, bpm_override=bpm)
|
||||
_require_source(parent.inp, inp, rebuild_policy.plan(parent.inp, inp))
|
||||
return _enqueue(new_id, inp, rebuilt_from=job_id)
|
||||
_require_source(parent.inp, inp, rebuild_policy.earliest(
|
||||
rebuild_policy.plan(parent.inp, inp), force_from))
|
||||
return _enqueue(new_id, inp, rebuilt_from=job_id, force_stage=force_from)
|
||||
|
||||
@app.get("/api/stages")
|
||||
def list_stages():
|
||||
@@ -305,6 +370,53 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
"video_filename": inp.video_path.name if inp.video_path else None,
|
||||
"cover_filename": inp.cover_path.name if inp.cover_path else None}
|
||||
|
||||
@app.delete("/api/jobs/{job_id}", status_code=204)
|
||||
def delete_job(job_id: str):
|
||||
with lock:
|
||||
job = jobs.get(job_id)
|
||||
if job is None:
|
||||
raise HTTPException(404)
|
||||
if job.status in ("queued", "running"):
|
||||
raise HTTPException(409, "job is still in progress")
|
||||
slug = song_slug(job.inp)
|
||||
if any(other.status in ("queued", "running") and song_slug(other.inp) == slug
|
||||
for other_id, other in jobs.items() if other_id != job_id):
|
||||
raise HTTPException(409, "another job for this song is in progress")
|
||||
jobs.pop(job_id, None)
|
||||
if job_id in order:
|
||||
order.remove(job_id)
|
||||
shutil.rmtree(cfg.work_dir / slug, ignore_errors=True)
|
||||
shutil.rmtree(cfg.output_dir / slug, ignore_errors=True)
|
||||
|
||||
def _resolve_media(job: Job, header: str) -> Path | None:
|
||||
"""The file a song folder's #MP3/#VIDEO/#COVER header points at."""
|
||||
slug = song_slug(job.inp)
|
||||
out = (cfg.output_dir / slug).resolve()
|
||||
txt = out / f"{slug}.txt"
|
||||
if not txt.exists():
|
||||
return None
|
||||
name = get_header(parse_ultrastar(txt.read_text(encoding="utf-8")), header)
|
||||
if not name:
|
||||
return None
|
||||
# a header names a file in the song folder and nothing else
|
||||
path = (out / name).resolve()
|
||||
if path.parent != out or not path.is_file():
|
||||
return None
|
||||
return path
|
||||
|
||||
@app.get("/api/jobs/{job_id}/media/{kind}")
|
||||
def get_media(job_id: str, kind: str):
|
||||
header = MEDIA_HEADERS.get(kind)
|
||||
if header is None:
|
||||
raise HTTPException(404)
|
||||
job = _finished_job(job_id)
|
||||
if job.status != "done":
|
||||
raise HTTPException(404)
|
||||
path = _resolve_media(job, header)
|
||||
if path is None:
|
||||
raise HTTPException(404)
|
||||
return FileResponse(path)
|
||||
|
||||
@app.get("/api/jobs/{job_id}/review")
|
||||
def get_review(job_id: str):
|
||||
job = _finished_job(job_id)
|
||||
@@ -335,7 +447,9 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
lines = [{"index": i, "text": " ".join(words), "dropped": i in dropped,
|
||||
"notes": by_line.get(i, []), "words": words_by_line.get(i, [])}
|
||||
for i, words in enumerate(ref.lines)]
|
||||
return {"lines": lines, "missing_spans": missing_spans}
|
||||
media = {kind: _resolve_media(job, header) is not None
|
||||
for kind, header in MEDIA_HEADERS.items()}
|
||||
return {"lines": lines, "missing_spans": missing_spans, "media": media}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
+93
-1
@@ -24,7 +24,7 @@ def test_local_video(tmp_path, monkeypatch):
|
||||
assert any("audio.wav" in str(a[-1]) for a in calls)
|
||||
|
||||
def test_youtube(tmp_path, monkeypatch):
|
||||
def fake_dl(url, dest):
|
||||
def fake_dl(url, dest, cfg, log):
|
||||
video = dest / "source_video.webm"
|
||||
video.write_bytes(b"v")
|
||||
thumb = dest / "source_video.jpg"
|
||||
@@ -83,3 +83,95 @@ def test_classify_picks_largest_video_when_multiple(tmp_path):
|
||||
(tmp_path / "source_video.mp4").write_bytes(b"much larger contents here")
|
||||
video, _ = acquire._classify_downloads(tmp_path)
|
||||
assert video.name == "source_video.mp4"
|
||||
|
||||
def test_classify_ignores_partials_left_by_a_failed_attempt(tmp_path):
|
||||
(tmp_path / "source_video.webm").write_bytes(b"v")
|
||||
(tmp_path / "source_video.f137.mp4.part").write_bytes(b"a much bigger partial")
|
||||
(tmp_path / "source_video.f251.webm.ytdl").write_bytes(b"resume state")
|
||||
video, _ = acquire._classify_downloads(tmp_path)
|
||||
assert video.name == "source_video.webm"
|
||||
|
||||
def test_ydl_opts_default_client_adds_no_player_client(tmp_path):
|
||||
cfg = Config(whisper_url="http://x/v1")
|
||||
opts = acquire._ydl_opts(cfg, tmp_path, "default", print)
|
||||
assert "youtube" not in opts["extractor_args"]
|
||||
assert opts["retries"] and opts["fragment_retries"] and opts["extractor_retries"]
|
||||
|
||||
def test_ydl_opts_named_client_and_pot_provider(tmp_path):
|
||||
cfg = Config(whisper_url="http://x/v1",
|
||||
ytdlp_pot_provider_url="http://pot:4416",
|
||||
ytdlp_cookies=tmp_path / "cookies.txt")
|
||||
opts = acquire._ydl_opts(cfg, tmp_path, "tv", print)
|
||||
assert opts["extractor_args"]["youtube"] == {"player_client": ["tv"]}
|
||||
assert opts["extractor_args"]["youtubepot-bgutilhttp"] == {
|
||||
"base_url": ["http://pot:4416"]}
|
||||
assert opts["cookiefile"] == str(tmp_path / "cookies.txt")
|
||||
|
||||
def _fake_yt_dlp(monkeypatch, behaviour):
|
||||
"""Install a stub yt_dlp whose download calls `behaviour(opts)`."""
|
||||
import sys
|
||||
import types
|
||||
seen = []
|
||||
|
||||
class FakeYoutubeDL:
|
||||
def __init__(self, opts):
|
||||
self.opts = opts
|
||||
def __enter__(self):
|
||||
return self
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
def extract_info(self, url, download):
|
||||
seen.append(self.opts["extractor_args"].get("youtube"))
|
||||
behaviour(self.opts)
|
||||
|
||||
mod = types.ModuleType("yt_dlp")
|
||||
mod.YoutubeDL = FakeYoutubeDL
|
||||
mod.version = types.SimpleNamespace(__version__="2026.08.19")
|
||||
monkeypatch.setitem(sys.modules, "yt_dlp", mod)
|
||||
return seen
|
||||
|
||||
def test_download_falls_through_to_the_next_client(tmp_path, monkeypatch):
|
||||
def behaviour(opts):
|
||||
client = opts["extractor_args"].get("youtube")
|
||||
if client is None:
|
||||
raise RuntimeError("HTTP Error 403: Forbidden")
|
||||
(tmp_path / "source_video.webm").write_bytes(b"v")
|
||||
seen = _fake_yt_dlp(monkeypatch, behaviour)
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=["default", "tv"])
|
||||
logged = []
|
||||
video, thumb = acquire.download_youtube("https://yt/x", tmp_path, cfg,
|
||||
logged.append)
|
||||
assert video.name == "source_video.webm"
|
||||
assert thumb is None
|
||||
assert seen == [None, {"player_client": ["tv"]}]
|
||||
assert any("403" in m for m in logged)
|
||||
|
||||
def test_download_reports_version_and_clients_when_all_fail(tmp_path, monkeypatch):
|
||||
import pytest
|
||||
from songtrace.pipeline import PipelineError
|
||||
def behaviour(opts):
|
||||
raise RuntimeError("HTTP Error 403: Forbidden")
|
||||
_fake_yt_dlp(monkeypatch, behaviour)
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=["default", "tv"])
|
||||
with pytest.raises(PipelineError) as e:
|
||||
acquire.download_youtube("https://yt/x", tmp_path, cfg, lambda m: None)
|
||||
assert "2026.08.19" in str(e.value)
|
||||
assert "default, tv" in str(e.value)
|
||||
assert "403" in str(e.value)
|
||||
|
||||
def test_download_with_no_configured_clients_still_tries_default(tmp_path, monkeypatch):
|
||||
seen = _fake_yt_dlp(
|
||||
monkeypatch, lambda opts: (tmp_path / "source_video.mp4").write_bytes(b"v"))
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=[])
|
||||
acquire.download_youtube("https://yt/x", tmp_path, cfg, lambda m: None)
|
||||
assert seen == [None]
|
||||
|
||||
def test_ydl_log_drops_debug_lines():
|
||||
logged = []
|
||||
log = acquire._YdlLog(logged.append)
|
||||
log.debug("[debug] internal detail")
|
||||
log.debug("[youtube] Extracting URL")
|
||||
log.warning("nsig extraction failed")
|
||||
log.error("HTTP Error 403: Forbidden")
|
||||
assert logged == ["[youtube] Extracting URL", "nsig extraction failed",
|
||||
"HTTP Error 403: Forbidden"]
|
||||
|
||||
+21
-3
@@ -49,19 +49,37 @@ def make_pipeline_inputs(tmp_path, cover=None):
|
||||
return inp, cfg
|
||||
|
||||
|
||||
def test_package_copies_compatible_video(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(package, "ffmpeg", lambda *a: Path(a[-1]).write_bytes(b"m"))
|
||||
def test_package_remuxes_compatible_video_without_reencoding(tmp_path, monkeypatch):
|
||||
calls = []
|
||||
monkeypatch.setattr(package, "ffmpeg",
|
||||
lambda *a: (calls.append(a), Path(a[-1]).write_bytes(b"m")))
|
||||
monkeypatch.setattr(package, "ffprobe_codec", lambda p: ("webm", "vp9"))
|
||||
ctx = make_ctx(tmp_path, cover=make_cover(tmp_path))
|
||||
package.package_stage().run(ctx)
|
||||
out = tmp_path / "out" / "A - T"
|
||||
assert (out / "A - T.txt").exists()
|
||||
assert (out / "A - T.mp3").exists()
|
||||
assert (out / "A - T.webm").read_bytes() == b"v"
|
||||
assert (out / "A - T.webm").exists()
|
||||
video_call = next(c for c in calls if c[-1].endswith("A - T.webm"))
|
||||
assert "-c" in video_call and "copy" in video_call, "the video was re-encoded"
|
||||
img = Image.open(out / "A - T.jpg")
|
||||
assert img.width == img.height
|
||||
|
||||
|
||||
@pytest.mark.parametrize("probe,name", [(("webm", "vp9"), "A - T.webm"),
|
||||
(("avi", "mpeg4"), "A - T.mp4")])
|
||||
def test_package_strips_the_videos_audio_track(tmp_path, monkeypatch, probe, name):
|
||||
"""#MP3 is the song's audio; a video that keeps its own plays as an echo."""
|
||||
calls = []
|
||||
monkeypatch.setattr(package, "ffmpeg",
|
||||
lambda *a: (calls.append(a), Path(a[-1]).write_bytes(b"m")))
|
||||
monkeypatch.setattr(package, "ffprobe_codec", lambda p: probe)
|
||||
ctx = make_ctx(tmp_path, cover=make_cover(tmp_path))
|
||||
package.package_stage().run(ctx)
|
||||
video_call = next(c for c in calls if c[-1].endswith(name))
|
||||
assert "-an" in video_call
|
||||
|
||||
|
||||
def test_package_reencodes_and_rewrites_video_header(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(package, "ffmpeg", lambda *a: Path(a[-1]).write_bytes(b"m"))
|
||||
monkeypatch.setattr(package, "ffprobe_codec", lambda p: ("avi", "mpeg4"))
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import SongInput, Stage, run_pipeline, song_slug
|
||||
from songtrace.pipeline import SongInput, Stage, read_warnings, run_pipeline, song_slug
|
||||
|
||||
def make_cfg(tmp_path):
|
||||
return Config(whisper_url="http://x/v1", output_dir=tmp_path / "out",
|
||||
@@ -46,3 +46,10 @@ def test_warnings_surfaced(tmp_path):
|
||||
st = Stage(name="n", outputs=lambda ctx: [ctx.work / "n.done"], run=run)
|
||||
result = run_pipeline(inp, cfg, log=lambda m: None, stages=[st])
|
||||
assert result.warnings == ["w1"]
|
||||
|
||||
def test_read_warnings_is_empty_without_a_notes_file(tmp_path):
|
||||
assert read_warnings(tmp_path) == []
|
||||
|
||||
def test_read_warnings_reads_the_notes_file(tmp_path):
|
||||
(tmp_path / "notes_sec.json").write_text(json.dumps({"warnings": ["w1", "w2"]}))
|
||||
assert read_warnings(tmp_path) == ["w1", "w2"]
|
||||
|
||||
+12
-1
@@ -5,7 +5,7 @@ import pytest
|
||||
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import STAGE_ORDER, PipelineError, SongInput
|
||||
from songtrace.rebuild import (FIELD_STAGE, changed_fields, plan,
|
||||
from songtrace.rebuild import (FIELD_STAGE, changed_fields, earliest, plan,
|
||||
relocate_work, superseded_output)
|
||||
|
||||
|
||||
@@ -124,3 +124,14 @@ def test_superseded_output_refuses_a_path_outside_the_output_root(tmp_path):
|
||||
def test_superseded_output_refuses_the_output_root_itself(tmp_path):
|
||||
root = tmp_path / "out"
|
||||
assert superseded_output(str(root), str(root / "A - T"), root) is None
|
||||
|
||||
|
||||
def test_earliest_picks_the_earlier_stage():
|
||||
assert earliest("package", "separate") == "separate"
|
||||
assert earliest("separate", "package") == "separate"
|
||||
|
||||
|
||||
def test_earliest_ignores_unset_stages():
|
||||
assert earliest(None, "grid") == "grid"
|
||||
assert earliest("grid", None) == "grid"
|
||||
assert earliest(None, None) is None
|
||||
|
||||
+612
-6
@@ -1,6 +1,7 @@
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
|
||||
@@ -583,11 +584,19 @@ def _submit_body(html):
|
||||
return html[start:html.index("\n});\n", start)]
|
||||
|
||||
|
||||
def _listener_body(html, prefix):
|
||||
start = html.index(prefix)
|
||||
return html[start:html.index("\n});\n", start)]
|
||||
|
||||
|
||||
def _rebuild_flaws(html):
|
||||
"""Ways rebuild mode could quietly stop working."""
|
||||
flaws, submit = [], _submit_body(html)
|
||||
if "rebuildOf ?" not in submit or "/rebuild" not in submit:
|
||||
flaws.append("the form always posts to /api/jobs, so edits create a new song")
|
||||
if "form.reset()" in submit or "exitRebuild()" in submit:
|
||||
flaws.append("a successful submission empties the form, so correcting "
|
||||
"or varying the song just sent means retyping it")
|
||||
|
||||
actions = _fn_body(html, "renderActions")
|
||||
if not re.search(r"if \(.*dataset\.jobId === job\.id.*\) return;", actions):
|
||||
@@ -595,12 +604,128 @@ def _rebuild_flaws(html):
|
||||
if '"failed"' not in actions or '"done"' not in actions:
|
||||
flaws.append("the action row does not key off the job status")
|
||||
|
||||
start = _fn_body(html, "startRebuild")
|
||||
if "/input" not in start or "form.elements[" not in start:
|
||||
load = _fn_body(html, "loadInputs")
|
||||
if "/input" not in load or "form.elements[" not in load:
|
||||
flaws.append("rebuild mode does not prefill the form from the job's input")
|
||||
|
||||
select = _fn_body(html, "selectJob")
|
||||
if "loadInputs(" not in select:
|
||||
flaws.append("selecting a queue entry does not bring its inputs back up")
|
||||
if "formDirty" not in select or "confirm(" not in select:
|
||||
flaws.append("selecting a queue entry discards unsaved edits without asking")
|
||||
|
||||
if "if (rebuildOf === id) exitRebuild();" not in _fn_body(html, "deleteJob"):
|
||||
flaws.append("deleting the loaded job leaves the form pointed at it")
|
||||
|
||||
# selecting a queue entry loads it, so the sidebar button is now the only
|
||||
# way back to a blank form
|
||||
create = _listener_body(html, 'createSong.addEventListener("click"')
|
||||
if "exitRebuild()" not in create:
|
||||
flaws.append("the queue's CREATE SONG button does not clear the form")
|
||||
if "formDirty" not in create or "confirm(" not in create:
|
||||
flaws.append("the queue's CREATE SONG button discards edits without asking")
|
||||
if "clearDetail(" not in create:
|
||||
flaws.append("CREATE SONG empties the form but leaves the previous "
|
||||
"song's review and player sitting under it")
|
||||
if "composing" not in create or "composing" not in _fn_body(html, "refreshJobs"):
|
||||
flaws.append("the blank page CREATE SONG made is selected back over by "
|
||||
"the next poll of the queue")
|
||||
|
||||
mode = _fn_body(html, "setFormMode")
|
||||
if 'querySelector("details").open' not in mode:
|
||||
flaws.append("prefilled genre/year/language/bpm stay hidden behind the "
|
||||
"closed Advanced summary, so they rebuild unseen")
|
||||
if "submitButton.disabled" not in _fn_body(html, "setSubmitLock"):
|
||||
flaws.append("the form lets you submit a rebuild the API will answer 409")
|
||||
return flaws
|
||||
|
||||
|
||||
def _rebuild_full_and_delete_flaws(html):
|
||||
flaws = []
|
||||
actions = _fn_body(html, "renderActions")
|
||||
for action in ('"rebuild-full"', '"delete"'):
|
||||
if action not in actions:
|
||||
flaws.append(f"{action} is not offered from renderActions")
|
||||
|
||||
rf = _fn_body(html, "rebuildFullJob")
|
||||
if "/rebuild-full" not in rf or 'method: "POST"' not in rf:
|
||||
flaws.append("rebuildFullJob does not POST to /rebuild-full")
|
||||
|
||||
d = _fn_body(html, "deleteJob")
|
||||
confirm_idx = d.find("confirm(")
|
||||
fetch_idx = d.find("fetch(")
|
||||
if confirm_idx == -1:
|
||||
flaws.append("deleteJob does not confirm before deleting")
|
||||
elif fetch_idx != -1 and confirm_idx > fetch_idx:
|
||||
flaws.append("deleteJob fetches before confirming")
|
||||
if 'method: "DELETE"' not in d:
|
||||
flaws.append("deleteJob does not send a DELETE request")
|
||||
return flaws
|
||||
|
||||
|
||||
def test_rebuild_full_and_delete_controls_are_wired_up():
|
||||
assert _rebuild_full_and_delete_flaws(_index_html()) == []
|
||||
|
||||
|
||||
def _empty_state_flaws(html):
|
||||
refresh = _fn_body(html, "refreshJobs")
|
||||
if "clearDetail(" not in refresh:
|
||||
return ["refreshJobs never resets the panel when no job is selected"]
|
||||
return []
|
||||
|
||||
|
||||
def test_review_panel_resets_when_no_job_is_selected():
|
||||
assert _empty_state_flaws(_index_html()) == []
|
||||
|
||||
|
||||
def _playback_flaws(html):
|
||||
"""Ways the review could stop tracking the media element."""
|
||||
flaws = []
|
||||
source = _fn_body(html, "setPlayerSource")
|
||||
if "media/audio" not in source:
|
||||
flaws.append("the player is not pointed at the job's packaged audio, "
|
||||
"which is the file the game plays and the notes track")
|
||||
if "media/video" not in source:
|
||||
flaws.append("the packaged video is never shown alongside the audio")
|
||||
|
||||
# the mp3 and the video carry the same recording; playing both is an echo
|
||||
if not re.search(r"<video[^>]*\bmuted\b", html):
|
||||
flaws.append("the background video is not muted, so its own audio "
|
||||
"plays on top of the mp3")
|
||||
pump = _fn_body(html, "pumpPlayback")
|
||||
if "slaveVideo(" not in pump:
|
||||
flaws.append("the picture is never pulled back onto the audio clock, "
|
||||
"so it drifts out of sync over a long song")
|
||||
|
||||
if "requestAnimationFrame" not in _fn_body(html, "pumpPlayback"):
|
||||
flaws.append("sync is not driven by the frame loop, so the playhead "
|
||||
"steps at timeupdate's ~4Hz instead of following the audio")
|
||||
|
||||
if "media.paused" not in _fn_body(html, "focusActive"):
|
||||
flaws.append("the view is scrolled even while paused, fighting a reader "
|
||||
"who scrolled away to look at something else")
|
||||
|
||||
sync = _fn_body(html, "syncPlayback")
|
||||
if "clearEntry(" not in sync:
|
||||
flaws.append("the line the playhead left keeps its highlight")
|
||||
clear = _fn_body(html, "clearEntry")
|
||||
for name in ("review-word-sung", "review-word-active", "review-line-active"):
|
||||
if name not in clear:
|
||||
flaws.append(f"{name} is never taken off a line the playhead left")
|
||||
|
||||
# every branch that gives up on rendering a review has to unload the media
|
||||
# too, or the previous song stays loaded and playable under the new one
|
||||
render = _fn_body(html, "renderLyricsReview")
|
||||
if render.count("resetPlayer()") < render.count("emptyNote("):
|
||||
flaws.append("a review the page cannot render leaves the previous "
|
||||
"song's media loaded in the player")
|
||||
return flaws
|
||||
|
||||
|
||||
def test_review_playback_tracks_the_media_element():
|
||||
assert _playback_flaws(_index_html()) == []
|
||||
|
||||
|
||||
# each mutation is a plausible regression; the checker above must reject all of them
|
||||
MUTATIONS = [
|
||||
(_focus_flaws, "queue: card moved unconditionally",
|
||||
@@ -629,6 +754,57 @@ MUTATIONS = [
|
||||
(_rebuild_flaws, "rebuild: action row rebuilt on every poll",
|
||||
"if (reviewActions.dataset.jobId === job.id && shown.join() === wanted.join()) return;",
|
||||
""),
|
||||
(_rebuild_full_and_delete_flaws, "delete: confirmation dialog removed",
|
||||
'if (!confirm("Delete this song? This removes all its files and cannot be undone.")) return;',
|
||||
""),
|
||||
(_rebuild_full_and_delete_flaws, "rebuild-full: no longer offered on a done job",
|
||||
'["rebuild-full", "delete"]', '["delete"]'),
|
||||
(_rebuild_flaws, "prefill: selecting a queue entry no longer loads its inputs",
|
||||
" if (reload) await loadInputs(id, JOBS.find(j => j.id === id)?.status);\n", ""),
|
||||
(_rebuild_flaws, "prefill: unsaved edits discarded without asking",
|
||||
' if (reload && formDirty && !confirm("Discard your unsaved edits to this form?")) return;\n',
|
||||
""),
|
||||
(_rebuild_flaws, "prefill: deleting the loaded job leaves the form pointed at it",
|
||||
" if (rebuildOf === id) exitRebuild();\n", ""),
|
||||
(_rebuild_flaws, "prefill: advanced fields stay hidden behind a closed summary",
|
||||
'form.querySelector("details").open =', "const unusedAdvancedOpen ="),
|
||||
(_rebuild_flaws, "prefill: an in-progress job can be submitted for rebuild anyway",
|
||||
"submitButton.disabled = locked;", ""),
|
||||
(_rebuild_flaws, "submit: a successful submission empties the form again",
|
||||
" rebuildOf = null;\n formDirty = false;\n setFormMode(null, null);\n refreshJobs();",
|
||||
" exitRebuild();\n refreshJobs();"),
|
||||
(_rebuild_flaws, "create: the queue button no longer clears the form",
|
||||
" exitRebuild();\n clearDetail(\"nothing selected\");\n", ""),
|
||||
(_rebuild_flaws, "create: the review is left behind under a blank form",
|
||||
' clearDetail("nothing selected");\n', ""),
|
||||
(_rebuild_flaws, "create: the next poll selects a song back into the blank page",
|
||||
" composing = true;\n", ""),
|
||||
(_rebuild_flaws, "create: the queue button discards edits without asking",
|
||||
' if (formDirty && !confirm("Discard your unsaved edits to this form?")) return;\n composing = true;',
|
||||
" composing = true;"),
|
||||
(_playback_flaws, "playback: sync falls back to the timeupdate event's ~4Hz",
|
||||
"syncFrame = media.paused ? null : requestAnimationFrame(pumpPlayback);",
|
||||
"syncFrame = null;"),
|
||||
(_playback_flaws, "playback: the view is scrolled even while paused",
|
||||
"if (!activeEntry || media.paused) return;", "if (!activeEntry) return;"),
|
||||
(_playback_flaws, "playback: the line the playhead left keeps its highlight",
|
||||
" if (activeEntry) clearEntry(activeEntry);\n", ""),
|
||||
(_playback_flaws, "playback: a word stays lit after the playhead moves on",
|
||||
'w.el.classList.remove("review-word-sung", "review-word-active");',
|
||||
'w.el.classList.remove("review-word-sung");'),
|
||||
(_playback_flaws, "playback: an unrenderable review leaves the old song loaded",
|
||||
" resetPlayer();\n", ""),
|
||||
(_playback_flaws, "playback: the player is not pointed at the packaged audio",
|
||||
'const src = available?.audio ? `/api/jobs/${job.id}/media/audio` : "";',
|
||||
'const src = "";'),
|
||||
(_playback_flaws, "playback: the background video plays its own audio too",
|
||||
'<video id="player-video" muted playsinline',
|
||||
'<video id="player-video" playsinline'),
|
||||
(_playback_flaws, "playback: the picture is never re-synced to the audio",
|
||||
" slaveVideo(false);\n", ""),
|
||||
(_empty_state_flaws, "empty state: clearDetail no longer wired into refreshJobs",
|
||||
"if (selectedId) refreshDetail();\n else clearDetail(composing ? \"nothing selected\" : undefined);",
|
||||
"if (selectedId) refreshDetail();"),
|
||||
]
|
||||
|
||||
|
||||
@@ -664,7 +840,7 @@ def test_rebuild_mode_reuses_the_one_form():
|
||||
|
||||
def test_index_has_rebuild_hooks():
|
||||
html = _index_html()
|
||||
for hook in ("review-actions", "review-lineage", "cancel-rebuild",
|
||||
for hook in ("review-actions", "review-lineage", "create-song",
|
||||
'data-keep="video"', 'data-keep="cover"'):
|
||||
assert hook in html, hook
|
||||
|
||||
@@ -846,7 +1022,7 @@ def test_rebuild_renames_the_work_dir_on_a_retitle(tmp_path):
|
||||
seen = []
|
||||
client, cfg = make_client(tmp_path, capture_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
(cfg.work_dir / "A - T" / "audio.wav").write_bytes(b"x")
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild",
|
||||
data=rebuild_form(title="T2")).json()["id"]
|
||||
@@ -859,7 +1035,7 @@ def test_rebuild_renames_the_work_dir_on_a_retitle(tmp_path):
|
||||
def test_rebuild_fails_the_job_on_a_work_dir_collision(tmp_path):
|
||||
client, cfg = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
(cfg.work_dir / "A - T2").mkdir(parents=True)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild",
|
||||
data=rebuild_form(title="T2")).json()["id"]
|
||||
@@ -887,7 +1063,7 @@ def test_rebuild_inherits_the_parents_source(tmp_path):
|
||||
files={"video": ("v.mp4", io.BytesIO(b"data"), "video/mp4")})
|
||||
parent = resp.json()["id"]
|
||||
wait_status(client, parent, "done")
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
for name in ("meta.json", "audio.wav"):
|
||||
(cfg.work_dir / "A - T" / name).write_bytes(b"x")
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild",
|
||||
@@ -973,6 +1149,59 @@ def test_rebuild_of_an_unknown_job_is_404(tmp_path):
|
||||
assert client.post("/api/jobs/nope/rebuild", data=rebuild_form()).status_code == 404
|
||||
|
||||
|
||||
def test_rebuild_full_forces_every_stage_from_the_first(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, capture_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild-full").json()["id"]
|
||||
wait_status(client, child, "done")
|
||||
assert seen[1]["force_from"] == STAGE_ORDER[0]
|
||||
assert seen[1]["inp"] == seen[0]["inp"]
|
||||
|
||||
|
||||
def test_rebuild_full_creates_a_linked_job(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
resp = client.post(f"/api/jobs/{parent}/rebuild-full")
|
||||
assert resp.status_code == 201
|
||||
child = resp.json()["id"]
|
||||
assert child != parent
|
||||
job = wait_status(client, child, "done")
|
||||
assert job["rebuilt_from"] == parent
|
||||
|
||||
|
||||
def test_rebuild_full_logs_that_it_is_a_forced_rebuild(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild-full").json()["id"]
|
||||
job = wait_status(client, child, "done")
|
||||
rebuild_lines = [l for l in job["log"] if l.startswith("[rebuild]")]
|
||||
assert len(rebuild_lines) == 1
|
||||
assert STAGE_ORDER[0] in rebuild_lines[0]
|
||||
|
||||
|
||||
def test_rebuild_full_of_an_unknown_job_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
assert client.post("/api/jobs/nope/rebuild-full").status_code == 404
|
||||
|
||||
|
||||
def test_rebuild_full_of_a_running_job_is_409(tmp_path):
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(inp, cfg, log=print, force_from=None, stages=None):
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, blocking)
|
||||
parent = client.post("/api/jobs", data=rebuild_form()).json()["id"]
|
||||
try:
|
||||
wait_status(client, parent, "running")
|
||||
assert client.post(f"/api/jobs/{parent}/rebuild-full").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, parent, "done")
|
||||
|
||||
|
||||
def slug_runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
out = cfg.output_dir / song_slug(inp)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
@@ -1021,6 +1250,76 @@ def test_rebuild_of_a_failed_parent_removes_nothing(tmp_path):
|
||||
assert (cfg.output_dir / "A - T" / "keep.txt").exists()
|
||||
|
||||
|
||||
def test_delete_removes_work_and_output_and_the_job(tmp_path):
|
||||
client, cfg = make_client(tmp_path, slug_runner)
|
||||
job_id = make_done_job(client)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
(cfg.work_dir / "A - T" / "audio.wav").write_bytes(b"x")
|
||||
assert (cfg.output_dir / "A - T").exists()
|
||||
|
||||
resp = client.delete(f"/api/jobs/{job_id}")
|
||||
|
||||
assert resp.status_code == 204
|
||||
assert not (cfg.work_dir / "A - T").exists()
|
||||
assert not (cfg.output_dir / "A - T").exists()
|
||||
assert client.get(f"/api/jobs/{job_id}").status_code == 404
|
||||
assert job_id not in [j["id"] for j in client.get("/api/jobs").json()]
|
||||
|
||||
|
||||
def test_delete_tolerates_a_work_dir_that_is_already_gone(tmp_path):
|
||||
client, cfg = make_client(tmp_path, slug_runner)
|
||||
job_id = make_done_job(client)
|
||||
shutil.rmtree(cfg.work_dir / "A - T")
|
||||
resp = client.delete(f"/api/jobs/{job_id}")
|
||||
assert resp.status_code == 204
|
||||
|
||||
|
||||
def test_delete_of_an_unknown_job_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
assert client.delete("/api/jobs/nope").status_code == 404
|
||||
|
||||
|
||||
def test_delete_of_a_running_job_is_409(tmp_path):
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(inp, cfg, log=print, force_from=None, stages=None):
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, blocking)
|
||||
parent = client.post("/api/jobs", data=rebuild_form()).json()["id"]
|
||||
try:
|
||||
wait_status(client, parent, "running")
|
||||
assert client.delete(f"/api/jobs/{parent}").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, parent, "done")
|
||||
assert client.get(f"/api/jobs/{parent}").status_code == 200
|
||||
|
||||
|
||||
def test_delete_of_a_job_with_an_active_rebuild_is_409(tmp_path):
|
||||
"""A done job's slug must not be deleted out from under a rebuild in flight."""
|
||||
release = threading.Event()
|
||||
calls = []
|
||||
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
calls.append(1)
|
||||
if len(calls) > 1:
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, runner)
|
||||
parent = make_done_job(client)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild-full").json()["id"]
|
||||
try:
|
||||
wait_status(client, child, "running")
|
||||
assert client.delete(f"/api/jobs/{parent}").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, child, "done")
|
||||
assert client.get(f"/api/jobs/{parent}").status_code == 200
|
||||
|
||||
|
||||
def test_a_plain_job_never_removes_anything(tmp_path):
|
||||
client, cfg = make_client(tmp_path, slug_runner)
|
||||
make_done_job(client)
|
||||
@@ -1030,6 +1329,54 @@ def test_a_plain_job_never_removes_anything(tmp_path):
|
||||
assert stray.exists()
|
||||
|
||||
|
||||
def test_discover_jobs_ignores_a_work_dir_without_a_sidecar(tmp_path):
|
||||
from songtrace.web import _discover_jobs
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
assert _discover_jobs(cfg) == []
|
||||
|
||||
|
||||
def test_discover_jobs_reconstructs_a_completed_song(tmp_path):
|
||||
from songtrace.web import _discover_jobs
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
client = TestClient(create_app(cfg, runner=slug_runner))
|
||||
make_done_job(client)
|
||||
|
||||
discovered = _discover_jobs(cfg)
|
||||
|
||||
assert len(discovered) == 1
|
||||
job = discovered[0]
|
||||
assert job.title == "T" and job.artist == "A" and job.status == "done"
|
||||
assert job.output_dir == str(cfg.output_dir / "A - T")
|
||||
assert job.error is None
|
||||
|
||||
|
||||
def test_discover_jobs_reconstructs_a_song_whose_output_is_missing_as_failed(tmp_path):
|
||||
from songtrace.web import _discover_jobs
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
client = TestClient(create_app(cfg, runner=slug_runner))
|
||||
make_done_job(client)
|
||||
shutil.rmtree(cfg.output_dir / "A - T")
|
||||
|
||||
discovered = _discover_jobs(cfg)
|
||||
|
||||
assert len(discovered) == 1
|
||||
assert discovered[0].status == "failed"
|
||||
assert discovered[0].error == "interrupted before completion"
|
||||
assert discovered[0].output_dir is None
|
||||
|
||||
|
||||
def test_discovered_jobs_are_visible_from_a_fresh_app(tmp_path):
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
client1 = TestClient(create_app(cfg, runner=slug_runner))
|
||||
make_done_job(client1)
|
||||
|
||||
client2 = TestClient(create_app(cfg, runner=slug_runner))
|
||||
jobs = client2.get("/api/jobs").json()
|
||||
assert len(jobs) == 1
|
||||
assert jobs[0]["title"] == "T" and jobs[0]["status"] == "done"
|
||||
|
||||
|
||||
def review_runner_factory(notes, dropped_lines=None, missing_spans=None, warnings=None,
|
||||
words=None):
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
@@ -1140,3 +1487,262 @@ def test_review_missing_when_pipeline_never_wrote_notes(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/review").status_code == 404
|
||||
|
||||
|
||||
# --- review playback -------------------------------------------------------
|
||||
|
||||
MEDIA_FILES = {"A - T.mp3": b"ID3audio", "A - T.mp4": b"\0\0\0\x18ftypvideo",
|
||||
"A - T.jpg": b"\xff\xd8cover"}
|
||||
MEDIA_HEADERS = [("TITLE", "T"), ("ARTIST", "A"), ("MP3", "A - T.mp3"),
|
||||
("COVER", "A - T.jpg"), ("VIDEO", "A - T.mp4")]
|
||||
|
||||
|
||||
def media_runner_factory(files=None, headers=None, write_txt=True):
|
||||
"""A run that leaves behind a song folder, the way package does."""
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
slug = song_slug(inp)
|
||||
out = cfg.output_dir / slug
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
for name, body in (MEDIA_FILES if files is None else files).items():
|
||||
(out / name).write_bytes(body)
|
||||
if write_txt:
|
||||
rows = [f"#{k}:{v}" for k, v in
|
||||
(MEDIA_HEADERS if headers is None else headers)]
|
||||
(out / f"{slug}.txt").write_text("\n".join(rows) + "\n: 0 4 0 la\nE\n",
|
||||
encoding="utf-8")
|
||||
return PipelineResult(output_dir=out)
|
||||
return runner
|
||||
|
||||
|
||||
def test_media_serves_the_files_the_song_txt_names(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
job_id = make_done_job(client)
|
||||
for kind, name in (("audio", "A - T.mp3"), ("video", "A - T.mp4"),
|
||||
("cover", "A - T.jpg")):
|
||||
resp = client.get(f"/api/jobs/{job_id}/media/{kind}")
|
||||
assert resp.status_code == 200, kind
|
||||
assert resp.content == MEDIA_FILES[name], kind
|
||||
|
||||
|
||||
def test_media_serves_whatever_the_header_was_rewritten_to(tmp_path):
|
||||
"""package transcodes to .mp4 and rewrites #VIDEO; the extension is not fixed."""
|
||||
files = {**MEDIA_FILES, "A - T.webm": b"transcoded"}
|
||||
headers = [(k, "A - T.webm" if k == "VIDEO" else v) for k, v in MEDIA_HEADERS]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(files, headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/video").content == b"transcoded"
|
||||
|
||||
|
||||
def test_media_supports_range_requests(tmp_path):
|
||||
"""Without 206 the browser cannot seek, which is the point of the player."""
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
job_id = make_done_job(client)
|
||||
resp = client.get(f"/api/jobs/{job_id}/media/video", headers={"Range": "bytes=0-3"})
|
||||
assert resp.status_code == 206
|
||||
assert resp.headers["content-range"] == f"bytes 0-3/{len(MEDIA_FILES['A - T.mp4'])}"
|
||||
assert resp.content == MEDIA_FILES["A - T.mp4"][:4]
|
||||
|
||||
|
||||
def test_media_of_an_unknown_kind_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/instrumental").status_code == 404
|
||||
|
||||
|
||||
def test_media_refuses_a_header_pointing_outside_the_song_folder(tmp_path):
|
||||
secret = tmp_path / "secret.txt"
|
||||
secret.write_text("not yours")
|
||||
headers = [(k, "../../secret.txt" if k == "MP3" else v) for k, v in MEDIA_HEADERS]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(headers=headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_refuses_an_absolute_header_path(tmp_path):
|
||||
secret = tmp_path / "secret.txt"
|
||||
secret.write_text("not yours")
|
||||
headers = [(k, str(secret) if k == "MP3" else v) for k, v in MEDIA_HEADERS]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(headers=headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_404s_when_the_named_file_is_missing(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory(files={}))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_404s_when_the_header_was_dropped(tmp_path):
|
||||
"""package strips #COVER when there is no usable cover image."""
|
||||
headers = [(k, v) for k, v in MEDIA_HEADERS if k != "COVER"]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(headers=headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/cover").status_code == 404
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 200
|
||||
|
||||
|
||||
def test_media_404s_when_there_is_no_song_txt(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory(write_txt=False))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_of_a_failed_job_is_404(tmp_path):
|
||||
def bad_runner(*a, **kw):
|
||||
raise PipelineError("boom")
|
||||
client, _ = make_client(tmp_path, bad_runner)
|
||||
job_id = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"}).json()["id"]
|
||||
wait_status(client, job_id, "failed")
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_of_a_running_job_is_409(tmp_path):
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(inp, cfg, log=print, force_from=None, stages=None):
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, blocking)
|
||||
job_id = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"}).json()["id"]
|
||||
try:
|
||||
wait_status(client, job_id, "running")
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, job_id, "done")
|
||||
|
||||
|
||||
def test_media_of_an_unknown_job_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
assert client.get("/api/jobs/nope/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_review_reports_which_media_the_player_can_load(tmp_path):
|
||||
notes = [{"start": 1.0, "duration": 0.4, "pitch": 0, "text": "hi",
|
||||
"low_confidence": False, "line_index": 0}]
|
||||
review = review_runner_factory(notes)
|
||||
media = media_runner_factory(headers=[(k, v) for k, v in MEDIA_HEADERS
|
||||
if k != "COVER"])
|
||||
|
||||
def runner(inp, cfg, **kw):
|
||||
media(inp, cfg, **kw)
|
||||
return review(inp, cfg, **kw)
|
||||
|
||||
client, _ = make_client(tmp_path, runner)
|
||||
job_id = make_done_job(client, lyrics="hi")
|
||||
body = client.get(f"/api/jobs/{job_id}/review").json()
|
||||
assert body["media"] == {"audio": True, "video": True, "cover": False}
|
||||
|
||||
|
||||
# --- restarting a rebuild from a chosen stage ------------------------------
|
||||
|
||||
def force_from_runner_factory(seen):
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
seen.append(force_from)
|
||||
out = cfg.output_dir / song_slug(inp)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
return PipelineResult(output_dir=out)
|
||||
return runner
|
||||
|
||||
|
||||
def rebuild_with(client, job_id, **extra):
|
||||
data = {"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"}
|
||||
data.update(extra)
|
||||
return client.post(f"/api/jobs/{job_id}/rebuild", data=data)
|
||||
|
||||
|
||||
def test_rebuild_without_edits_restarts_from_the_stage_asked_for(tmp_path):
|
||||
"""The point of the control: a change in songtrace invalidates no field."""
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, force_from="package")
|
||||
assert resp.status_code == 201
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "package"
|
||||
|
||||
|
||||
def test_rebuild_without_a_stage_still_plans_from_the_edit(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, lyrics="different words")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "align"
|
||||
|
||||
|
||||
def test_an_edit_wins_when_it_invalidates_more_than_the_stage_asked_for(tmp_path):
|
||||
"""A choice may widen what is redone, never narrow it below correctness."""
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, lyrics="different words",
|
||||
force_from="package")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "align"
|
||||
|
||||
|
||||
def test_the_stage_asked_for_wins_when_it_is_earlier_than_the_edit(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, title="Renamed", force_from="separate")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "separate"
|
||||
|
||||
|
||||
def test_rebuild_rejects_a_stage_that_is_not_in_the_pipeline(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, force_from="deechoing")
|
||||
assert resp.status_code == 400
|
||||
assert "deechoing" in resp.json()["detail"]
|
||||
|
||||
|
||||
def test_a_blank_stage_is_the_same_as_not_asking(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, force_from="")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] is None
|
||||
|
||||
|
||||
def test_rebuild_from_scratch_still_restarts_at_the_first_stage(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = client.post(f"/api/jobs/{parent}/rebuild-full")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == STAGE_ORDER[0]
|
||||
|
||||
|
||||
def test_the_restart_control_offers_every_stage():
|
||||
html = _index_html()
|
||||
assert 'name="force_from"' in html
|
||||
stages = _fn_body(html, "loadStages")
|
||||
assert "force_from" in stages and "STAGES.map" in stages
|
||||
mode = _fn_body(html, "setFormMode")
|
||||
assert "restart-field" in mode, "the control is shown outside rebuild mode"
|
||||
|
||||
|
||||
def test_a_textareas_crlf_does_not_read_as_a_lyrics_edit(tmp_path):
|
||||
"""Browsers submit textareas with CRLF; the stored lyrics use LF."""
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client, lyrics="one\ntwo")
|
||||
resp = rebuild_with(client, parent, lyrics="one\r\ntwo")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] is None, "the rebuild redid the alignment for nothing"
|
||||
body = client.get(f"/api/jobs/{resp.json()['id']}/input").json()
|
||||
assert body["lyrics"] == "one\ntwo"
|
||||
|
||||
|
||||
def test_submitting_crlf_lyrics_stores_them_with_lf(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
job_id = make_done_job(client, lyrics="one\r\ntwo")
|
||||
assert client.get(f"/api/jobs/{job_id}/input").json()["lyrics"] == "one\ntwo"
|
||||
|
||||
Reference in New Issue
Block a user