Fix YouTube 403s: install yt-dlp's YouTube extras and retry across player clients #8

2026-09-25T18:25:00-07:00

dejima-agent commented

2026-09-25 18:25:00 -07:00

Summary generated by LLM agent

The cause

yt-dlp's own pyproject.toml declares dependencies = [] and puts everything it actually needs into extras. Our requirement was a bare "yt-dlp", so the image got the extractor and nothing else. The relevant omission is yt-dlp-ejs (from the default extra) plus a JavaScript engine to run it: yt-dlp needs both to solve YouTube's player challenges. Without them it does not fail loudly at extraction time, it quietly selects formats whose media URLs YouTube then rejects, which surfaces exactly as ERROR: unable to download video data: HTTP Error 403: Forbidden.

Verified against yt-dlp 2026.08.19. With the bare install the debug header reports Optional libraries: none and JS runtimes: none; with the extras it reports JS runtimes: deno-2.9.7 and JS Challenge Providers: ... deno.

Worth noting separately: the deployed image also predates stable 2026.08.19, which is the release that fixed the widespread August 2026 android_vr 403 regression. So the running container had two independent reasons to 403.

The fix

The requirement becomes yt-dlp[default,curl-cffi,deno]. The deno extra installs the runtime as a platform wheel that drops a binary into /usr/local/bin, so nothing changes in the Dockerfile and no apt packages are added. curl-cffi is what makes --impersonate targets available.

Resilience around it

The root cause will recur, because YouTube changes its defences every few weeks and an image pinned at build time does not. So the stage no longer makes one attempt and gives up:

The download is retried against each player client in ytdlp_clients, defaulting to ["default", "tv"]. default is yt-dlp's own choice, which is the best-maintained option; tv is the fallback because it is one of the few clients that does not require a PO token for media requests. A client YouTube has broken is now skipped automatically rather than failing the job.
retries, fragment_retries and extractor_retries are raised, which covers the known class of intermittent 403s on individual fragments.
yt-dlp's own log lines are routed into the job log (progress suppressed, [debug] lines dropped). This is the difference between a 403 you can diagnose and one you cannot: a PO-token demand, a rate limit and a signature-extraction failure all present as 403 at the top level but say different things in yt-dlp's output.
The final error names the running yt-dlp version and the clients tried, since a stale yt-dlp is the most likely explanation and was the actual one here.

Three new config fields let an operator work around a future block without a code change: ytdlp_cookies (Netscape cookie jar), ytdlp_impersonate (browser TLS fingerprint, e.g. "chrome" — off by default, since yt-dlp warns that forcing it for all requests costs speed and stability), and ytdlp_pot_provider_url for a bgutil-ytdlp-pot-provider sidecar. The README gains a troubleshooting section ordering the causes by likelihood, including how to hotfix a running container versus rebuild it.

One incidental bug fixed along the way: _classify_downloads globbed source_video.* and treated anything without an image extension as a candidate video, so a leftover source_video.f137.mp4.part counted as a video and could win the largest-file comparison. That was already reachable after any interrupted download; retrying across clients makes partial files much more likely to be lying around, so it is fixed here rather than later.

Testing

Added coverage for the retry ladder (falls through to the next client and returns on success; reports version, clients tried and last error when all fail; an empty ytdlp_clients list still attempts the default), for _ydl_opts construction across the config permutations, for the .part/.ytdl exclusion, and for the log adapter's [debug] filtering. tests/test_acquire.py and tests/test_config.py pass, 21 tests.

Full suite: 279 passed, 1 skipped. tests/test_audio_io.py and tests/test_separate.py were not run — they import torch, which is not installable in the environment I had. Neither touches this change. tests/test_web.py::test_upload_dir_cleaned_up_after_success is flaky on timing and fails roughly one run in three in isolation; it is pre-existing and unrelated, as this branch does not touch web.py.

The generated options dict was validated against real yt-dlp 2026.08.19 by constructing a YoutubeDL from it for both clients, including the ImpersonateTarget. What I could not test is an actual YouTube download: outbound network here is restricted to an allowlist that does not include YouTube, so the end-to-end confirmation that the 403 is gone still needs to happen on the deployed host. docker compose build --pull songtrace and then re-running the failed job is the check.

**Summary generated by LLM agent** ## The cause `yt-dlp`'s own `pyproject.toml` declares `dependencies = []` and puts everything it actually needs into extras. Our requirement was a bare `"yt-dlp"`, so the image got the extractor and nothing else. The relevant omission is `yt-dlp-ejs` (from the `default` extra) plus a JavaScript engine to run it: yt-dlp needs both to solve YouTube's player challenges. Without them it does not fail loudly at extraction time, it quietly selects formats whose media URLs YouTube then rejects, which surfaces exactly as `ERROR: unable to download video data: HTTP Error 403: Forbidden`. Verified against yt-dlp 2026.08.19. With the bare install the debug header reports `Optional libraries: none` and `JS runtimes: none`; with the extras it reports `JS runtimes: deno-2.9.7` and `JS Challenge Providers: ... deno`. Worth noting separately: the deployed image also predates stable 2026.08.19, which is the release that fixed the widespread August 2026 `android_vr` 403 regression. So the running container had two independent reasons to 403. ## The fix The requirement becomes `yt-dlp[default,curl-cffi,deno]`. The `deno` extra installs the runtime as a platform wheel that drops a binary into `/usr/local/bin`, so nothing changes in the Dockerfile and no apt packages are added. `curl-cffi` is what makes `--impersonate` targets available. ## Resilience around it The root cause will recur, because YouTube changes its defences every few weeks and an image pinned at build time does not. So the stage no longer makes one attempt and gives up: - The download is retried against each player client in `ytdlp_clients`, defaulting to `["default", "tv"]`. `default` is yt-dlp's own choice, which is the best-maintained option; `tv` is the fallback because it is one of the few clients that does not require a PO token for media requests. A client YouTube has broken is now skipped automatically rather than failing the job. - `retries`, `fragment_retries` and `extractor_retries` are raised, which covers the known class of intermittent 403s on individual fragments. - yt-dlp's own log lines are routed into the job log (progress suppressed, `[debug]` lines dropped). This is the difference between a 403 you can diagnose and one you cannot: a PO-token demand, a rate limit and a signature-extraction failure all present as 403 at the top level but say different things in yt-dlp's output. - The final error names the running yt-dlp version and the clients tried, since a stale yt-dlp is the most likely explanation and was the actual one here. Three new config fields let an operator work around a future block without a code change: `ytdlp_cookies` (Netscape cookie jar), `ytdlp_impersonate` (browser TLS fingerprint, e.g. `"chrome"` — off by default, since yt-dlp warns that forcing it for all requests costs speed and stability), and `ytdlp_pot_provider_url` for a `bgutil-ytdlp-pot-provider` sidecar. The README gains a troubleshooting section ordering the causes by likelihood, including how to hotfix a running container versus rebuild it. One incidental bug fixed along the way: `_classify_downloads` globbed `source_video.*` and treated anything without an image extension as a candidate video, so a leftover `source_video.f137.mp4.part` counted as a video and could win the largest-file comparison. That was already reachable after any interrupted download; retrying across clients makes partial files much more likely to be lying around, so it is fixed here rather than later. ## Testing Added coverage for the retry ladder (falls through to the next client and returns on success; reports version, clients tried and last error when all fail; an empty `ytdlp_clients` list still attempts the default), for `_ydl_opts` construction across the config permutations, for the `.part`/`.ytdl` exclusion, and for the log adapter's `[debug]` filtering. `tests/test_acquire.py` and `tests/test_config.py` pass, 21 tests. Full suite: 279 passed, 1 skipped. `tests/test_audio_io.py` and `tests/test_separate.py` were not run — they import torch, which is not installable in the environment I had. Neither touches this change. `tests/test_web.py::test_upload_dir_cleaned_up_after_success` is flaky on timing and fails roughly one run in three in isolation; it is pre-existing and unrelated, as this branch does not touch `web.py`. The generated options dict was validated against real yt-dlp 2026.08.19 by constructing a `YoutubeDL` from it for both clients, including the `ImpersonateTarget`. What I could not test is an actual YouTube download: outbound network here is restricted to an allowlist that does not include YouTube, so the end-to-end confirmation that the 403 is gone still needs to happen on the deployed host. `docker compose build --pull songtrace` and then re-running the failed job is the check.

dejima-agent added 1 commit 2026-09-25 18:25:01 -07:00

acquire: install yt-dlp's YouTube extras and retry across player clients 211c0ac6a5

yt-dlp declares `dependencies = []` and puts everything in extras, so the
bare `yt-dlp` requirement installed it with no yt-dlp-ejs and no JavaScript
runtime. Without those it cannot solve YouTube's player challenges and falls
back to media URLs that YouTube rejects with HTTP 403, which is the failure
seen when adding a song. Require `yt-dlp[default,curl-cffi,deno]` instead;
the deno extra ships the JS runtime as a wheel, so the image needs no extra
apt packages.

Around that, make the stage survive the next time YouTube changes something:
the download is retried against each player client in `ytdlp_clients`
(default `["default", "tv"]`, the latter not needing a PO token) before
giving up, yt-dlp's retry counts are raised, and its own log lines are routed
into the job log so a PO-token demand can be told apart from a rate limit.
The final error names the running yt-dlp version, since a stale one is the
usual cause.

Cookies, TLS impersonation and a bgutil PO token provider URL are now config
fields, so an operator can work around a block without a code change.
`_classify_downloads` skips .part/.ytdl scratch files, which retries make far
more likely to be lying around.

branderson approved these changes 2026-09-25 18:32:15 -07:00

branderson merged commit fc8f5c65ac into main

2026-09-25 18:32:20 -07:00

branderson referenced this issue from a commit

2026-10-15 14:17:07 -07:00

Merge pull request 'Fix YouTube 403s: install yt-dlp's YouTube extras and retry across player clients' (#8) from ytdlp-resilience into main

dejima-agent referenced this issue from a commit

2026-10-15 17:44:36 -07:00

Merge pull request 'Fix YouTube 403s: install yt-dlp's YouTube extras and retry across player clients' (#8) from ytdlp-resilience into main

Sign in to join this conversation.