Compare commits
55
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4272189bca | ||
|
|
936641e13c | ||
|
|
793817ce42 | ||
|
|
f3125370d3 | ||
|
|
6d297a8bfc | ||
|
|
4af78fdd1d | ||
|
|
cb485d3c6b | ||
|
|
f187bfd6c0 | ||
|
|
22c7f6d03f | ||
|
|
7c4359eb39 | ||
|
|
ba8bf9f4af | ||
|
|
be67e01943 | ||
|
|
b5850042be | ||
|
|
55e8297672 | ||
|
|
a7622c2d01 | ||
|
|
c489dd5742 | ||
|
|
f1b967b487 | ||
|
|
e141cc5773 | ||
|
|
92948b2ea7 | ||
|
|
c8c3d941ca | ||
|
|
7b8fce5b6e | ||
|
|
cb10400a21 | ||
|
|
98e4f9532f | ||
|
|
77bebcf9e1 | ||
|
|
986f26b8ed | ||
|
|
d557ac6386 | ||
|
|
5023ea69d3 | ||
|
|
0599f6b665 | ||
|
|
2debfb171c | ||
|
|
ac904cb09f | ||
|
|
8e89b08571 | ||
|
|
6a98ea0f21 | ||
|
|
903e83dbcc | ||
|
|
a4e0b45953 | ||
|
|
ffd662fb38 | ||
|
|
43f0554796 | ||
|
|
ebb7fc4433 | ||
|
|
cf1c7410ab | ||
|
|
5bc4855264 | ||
|
|
a650c2f7ec | ||
|
|
283da7e1a9 | ||
|
|
292fa25d5e | ||
|
|
0eaab6aeed | ||
|
|
7a48942f1c | ||
|
|
6c15c3d8f1 | ||
|
|
d2b7986cb9 | ||
|
|
3f90f4857f | ||
|
|
5553614fdf | ||
|
|
8a87534c5d | ||
|
|
322f8eb98b | ||
|
|
a3078bb2fc | ||
|
|
bd1b4aeb6e | ||
|
|
f2a97f4f10 | ||
|
|
fca9e874f3 | ||
|
|
bd5593c3ed |
+2
-2
@@ -11,9 +11,9 @@ COPY pyproject.toml ./
|
||||
COPY src ./src
|
||||
RUN pip install --no-cache-dir .
|
||||
|
||||
ENV KARAOKE_MAKER_CONFIG=/config/config.toml \
|
||||
ENV SONGTRACE_CONFIG=/config/config.toml \
|
||||
TORCH_HOME=/data/models \
|
||||
XDG_CACHE_HOME=/data/models/cache
|
||||
|
||||
EXPOSE 8000
|
||||
CMD ["karaoke-maker", "serve"]
|
||||
CMD ["songtrace", "serve"]
|
||||
|
||||
@@ -0,0 +1,661 @@
|
||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
||||
Version 3, 19 November 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU Affero General Public License is a free, copyleft license for
|
||||
software and other kinds of works, specifically designed to ensure
|
||||
cooperation with the community in the case of network server software.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
our General Public Licenses are intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
Developers that use our General Public Licenses protect your rights
|
||||
with two steps: (1) assert copyright on the software, and (2) offer
|
||||
you this License which gives you legal permission to copy, distribute
|
||||
and/or modify the software.
|
||||
|
||||
A secondary benefit of defending all users' freedom is that
|
||||
improvements made in alternate versions of the program, if they
|
||||
receive widespread use, become available for other developers to
|
||||
incorporate. Many developers of free software are heartened and
|
||||
encouraged by the resulting cooperation. However, in the case of
|
||||
software used on network servers, this result may fail to come about.
|
||||
The GNU General Public License permits making a modified version and
|
||||
letting the public access it on a server without ever releasing its
|
||||
source code to the public.
|
||||
|
||||
The GNU Affero General Public License is designed specifically to
|
||||
ensure that, in such cases, the modified source code becomes available
|
||||
to the community. It requires the operator of a network server to
|
||||
provide the source code of the modified version running there to the
|
||||
users of that server. Therefore, public use of a modified version, on
|
||||
a publicly accessible server, gives the public access to the source
|
||||
code of the modified version.
|
||||
|
||||
An older license, called the Affero General Public License and
|
||||
published by Affero, was designed to accomplish similar goals. This is
|
||||
a different license, not a version of the Affero GPL, but Affero has
|
||||
released a new version of the Affero GPL which permits relicensing under
|
||||
this license.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, if you modify the
|
||||
Program, your modified version must prominently offer all users
|
||||
interacting with it remotely through a computer network (if your version
|
||||
supports such interaction) an opportunity to receive the Corresponding
|
||||
Source of your version by providing access to the Corresponding Source
|
||||
from a network server at no charge, through some standard or customary
|
||||
means of facilitating copying of software. This Corresponding Source
|
||||
shall include the Corresponding Source for any work covered by version 3
|
||||
of the GNU General Public License that is incorporated pursuant to the
|
||||
following paragraph.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the work with which it is combined will remain governed by version
|
||||
3 of the GNU General Public License.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU Affero General Public License from time to time. Such new versions
|
||||
will be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU Affero General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU Affero General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU Affero General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published
|
||||
by the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If your software can interact with users remotely through a computer
|
||||
network, you should also make sure that it provides a way for users to
|
||||
get its source. For example, if your program is a web application, its
|
||||
interface could display a "Source" link that leads users to an archive
|
||||
of the code. There are many ways you could offer source, and different
|
||||
solutions will be better for different programs; see section 13 for the
|
||||
specific requirements.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
@@ -1,17 +1,22 @@
|
||||
# karaoke-maker
|
||||
# songtrace
|
||||
|
||||
Turn a song and its lyrics into a playable UltraStar chart.
|
||||
|
||||
A CLI and minimal web UI that turns a song — title, artist, lyrics you supply,
|
||||
and a YouTube URL or local video file — into a playable
|
||||
[UltraStar](https://usdx.eu/format/)-format song folder for
|
||||
[Melody Mania](https://melodymania.net/), a commercial UltraStar Deluxe fork.
|
||||
[UltraStar](https://usdx.eu/format/)-format song folder. Also compatible with
|
||||
[Melody Mania](https://melodymania.org/).
|
||||
|
||||

|
||||
|
||||
The pipeline:
|
||||
|
||||
1. **acquire** — download audio/video via `yt-dlp` (or use a local video file)
|
||||
2. **separate** — split vocals/instrumental with Demucs
|
||||
3. **transcribe** — send the vocal stem to your own intranet OpenAI-compatible
|
||||
Whisper server, as a sanity check that the supplied lyrics match the audio
|
||||
and to locate vocal sections
|
||||
3. **transcribe** — send the vocal stem to your own intranet Whisper server
|
||||
(OpenAI-compatible, or [onerahmet/openai-whisper-asr-webservice](https://github.com/ahmetoner/whisper-asr-webservice)),
|
||||
as a sanity check that the supplied lyrics match the audio and to locate
|
||||
vocal sections
|
||||
4. **align** — forced-align the *supplied* reference lyrics against the vocal
|
||||
stem with wav2vec2 (torchaudio's MMS_FA) to get word timings
|
||||
5. **pitch** — track pitch on the vocal stem with torchcrepe
|
||||
@@ -26,16 +31,37 @@ output.
|
||||
**The output is a best-effort draft**, not a finished song. Every run ends by
|
||||
printing the low-confidence regions it found (e.g.
|
||||
`3 low-confidence notes near 0:42–0:47`) — open the result in an UltraStar
|
||||
editor such as [Yass](http://www.yass-hq.de/) or
|
||||
editor such as [Yass](https://www.yass-along.com/) or
|
||||
[USDX](https://usdx.eu/) and touch those spots up before playing.
|
||||
|
||||
## Reviewing a song
|
||||
|
||||
The web UI plays the finished song folder back against the review, so you can
|
||||
check the timing without loading it into a game first. Pick a song in the
|
||||
queue and the review panel gets the packaged video with a transport above the
|
||||
lyrics; as it plays, the line being sung is highlighted, words light up as
|
||||
they are reached, and a playhead sweeps each line against its notes. Click any
|
||||
line or word to play from there.
|
||||
|
||||
Playback is modelled on the game: the `#MP3` is the audio and the clock, and
|
||||
the `#VIDEO` is muted background slaved to it. It also uses the same seconds
|
||||
the notes were generated from, so a word that lights up late in the browser is
|
||||
late in the game too.
|
||||
|
||||
The packaged video has its audio track stripped for the same reason. It is the
|
||||
same recording as the `#MP3`, so a player that does not mute the video plays
|
||||
both at once, a few frames apart, which is heard as an echo. Songs built
|
||||
before this change still have it — check with `ffprobe <song>.mp4` for an
|
||||
audio stream, and rebuild with **Restart from** set to `package` to clear it
|
||||
without re-downloading.
|
||||
|
||||
## Configuration
|
||||
|
||||
Config is TOML. Resolution order:
|
||||
|
||||
1. `--config <path>` passed explicitly on the command line
|
||||
2. the `KARAOKE_MAKER_CONFIG` environment variable
|
||||
3. `~/.config/karaoke-maker/config.toml`
|
||||
2. the `SONGTRACE_CONFIG` environment variable
|
||||
3. `~/.config/songtrace/config.toml`
|
||||
|
||||
Copy the example and fill in your Whisper server:
|
||||
|
||||
@@ -45,8 +71,17 @@ $EDITOR config.toml # set whisper_url at minimum
|
||||
```
|
||||
|
||||
See `config.example.toml` for every available field (`whisper_url`,
|
||||
`whisper_api_key`, `whisper_model`, `output_dir`, `work_dir`, `creator`,
|
||||
`demucs_model`, `crepe_model`, `device`, `bind_host`, `port`).
|
||||
`whisper_api_key`, `whisper_model`, `whisper_api`, `output_dir`, `work_dir`,
|
||||
`creator`, `demucs_model`, `crepe_model`, `device`, `bind_host`, `port`,
|
||||
`ytdlp_clients`, `ytdlp_cookies`, `ytdlp_impersonate`,
|
||||
`ytdlp_pot_provider_url`, `ytdlp_version`).
|
||||
|
||||
`whisper_api` selects the wire format: `"openai"` (default) speaks the
|
||||
OpenAI-compatible `POST {whisper_url}/audio/transcriptions` API; `"asr-webservice"`
|
||||
speaks [onerahmet/openai-whisper-asr-webservice](https://github.com/ahmetoner/whisper-asr-webservice)'s
|
||||
`POST {whisper_url}/asr` API instead (point `whisper_url` at that server's base
|
||||
URL with no `/v1` suffix — `whisper_model`/`whisper_api_key` are unused for
|
||||
this backend, since the model is fixed by how that server was started).
|
||||
|
||||
## Docker
|
||||
|
||||
@@ -63,7 +98,7 @@ container recreation.
|
||||
Run the CLI inside the container instead of the web UI:
|
||||
|
||||
```sh
|
||||
docker compose run --rm karaoke-maker karaoke-maker make \
|
||||
docker compose run --rm songtrace songtrace make \
|
||||
--title "Song Title" --artist "Artist Name" \
|
||||
--lyrics /data/lyrics.txt --youtube https://youtu.be/...
|
||||
```
|
||||
@@ -73,21 +108,45 @@ Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/)
|
||||
installed on the host; `docker-compose.yml` requests one GPU via the
|
||||
`nvidia` device driver.
|
||||
|
||||
## Sharing the GPU
|
||||
|
||||
`songtrace serve` is long-lived but only wants the GPU in bursts, so it hands
|
||||
it back after every stage rather than holding it for the life of the process.
|
||||
Three stages load a model — `separate` (Demucs), `align` (wav2vec2) and
|
||||
`pitch` (torchcrepe) — and each one's weights are dropped as soon as that
|
||||
stage finishes, followed by `torch.cuda.empty_cache()` to return the
|
||||
allocator's freed blocks to the driver instead of keeping them for reuse. This
|
||||
matters most between `separate` and `align`: `transcribe` sits between them
|
||||
waiting on your Whisper server, which can be minutes of a GPU nothing is
|
||||
using.
|
||||
|
||||
Stages that gave memory back say so in the job log:
|
||||
|
||||
```
|
||||
[separate] released 3704 MiB of GPU memory
|
||||
```
|
||||
|
||||
The cost is reloading each model once per job instead of once per process,
|
||||
which is seconds against stages that run for minutes. What this does not
|
||||
release is the CUDA context itself, a few hundred MB that belongs to the
|
||||
process until it exits; freeing that would mean running each stage in a
|
||||
subprocess. If you need every last MB back, stop the container between runs.
|
||||
|
||||
## Bare-metal
|
||||
|
||||
```sh
|
||||
pip install -e .
|
||||
karaoke-maker make --help
|
||||
songtrace make --help
|
||||
```
|
||||
|
||||
```
|
||||
karaoke-maker make --title TITLE --artist ARTIST --lyrics LYRICS_FILE \
|
||||
songtrace make --title TITLE --artist ARTIST --lyrics LYRICS_FILE \
|
||||
(--youtube URL | --video PATH) \
|
||||
[--genre GENRE] [--year YEAR] [--language LANGUAGE] [--cover COVER_IMAGE] \
|
||||
[--bpm BPM] [--force-from STAGE] [--config CONFIG]
|
||||
```
|
||||
|
||||
Run the web UI instead with `karaoke-maker serve [--config CONFIG]`.
|
||||
Run the web UI instead with `songtrace serve [--config CONFIG]`.
|
||||
|
||||
## Re-running a stage
|
||||
|
||||
@@ -98,9 +157,82 @@ stage and everything after it, reusing the cached output of everything
|
||||
before it:
|
||||
|
||||
```sh
|
||||
karaoke-maker make --title "..." --artist "..." --lyrics lyrics.txt \
|
||||
songtrace make --title "..." --artist "..." --lyrics lyrics.txt \
|
||||
--youtube https://youtu.be/... --force-from align
|
||||
```
|
||||
|
||||
Valid stages, in pipeline order: `acquire`, `separate`, `transcribe`,
|
||||
`align`, `pitch`, `notes`, `grid`, `package`.
|
||||
|
||||
In the web UI this is what selecting a queue entry does: its title, artist,
|
||||
lyrics and the rest come back up in the form, so you can adjust them and hit
|
||||
REBUILD SONG. Only the stages your edit actually invalidates are rerun —
|
||||
fixing a line of lyrics realigns without re-downloading or re-separating.
|
||||
CREATE SONG, at the top of the queue, clears the page to start a fresh one.
|
||||
|
||||
That planning works off the fields you changed, which cannot see a change in
|
||||
songtrace itself: after an upgrade, a rebuild of an older song finds every
|
||||
stage cached and does nothing. **Restart from** on the rebuild form is the way
|
||||
to redo a stage anyway — pick `package` to repackage without re-downloading.
|
||||
It can only widen what is rerun, so an edit that invalidates an earlier stage
|
||||
still wins.
|
||||
|
||||
## YouTube download failures
|
||||
|
||||
`HTTP Error 403: Forbidden` from the acquire stage means YouTube served the
|
||||
page but refused the media URL. In rough order of likelihood:
|
||||
|
||||
1. **yt-dlp is stale.** YouTube changes its defences every few weeks, so
|
||||
songtrace keeps yt-dlp current itself. Once a day, and again whenever a
|
||||
download fails outright, it checks PyPI for a newer stable release,
|
||||
installs it under `$XDG_CACHE_HOME/songtrace/` (`/data/models/cache` in the
|
||||
container), and tries it before the copy installed with songtrace. That
|
||||
copy is never modified and is always the fallback, so a release that breaks
|
||||
compatibility costs a retry rather than the download. The job log names
|
||||
the yt-dlp version behind each attempt.
|
||||
|
||||
`ytdlp_version` overrides this: `"bundled"` uses only the installed copy,
|
||||
and a release number such as `"2026.08.19"` pins that release instead of
|
||||
the newest.
|
||||
|
||||
2. **Missing JS runtime.** yt-dlp needs `yt-dlp-ejs` plus a JavaScript engine
|
||||
to solve YouTube's player challenges; without them it silently falls back to
|
||||
formats that 403. The `[default,curl-cffi,deno]` extras in `pyproject.toml`
|
||||
cover this, for both the installed copy and the ones fetched at runtime.
|
||||
Check with `docker compose exec songtrace yt-dlp -v --simulate <url>` — the debug
|
||||
header must show a `JS runtimes:` line that is not `none`.
|
||||
|
||||
3. **The IP is rate-limited or flagged.** Datacenter and VPN ranges get
|
||||
throttled hardest. Try from another network before changing config.
|
||||
|
||||
4. **A PO token is required.** If the log says so explicitly, run
|
||||
[bgutil-ytdlp-pot-provider](https://github.com/Brainicism/bgutil-ytdlp-pot-provider)
|
||||
as a sidecar, install its plugin into the image, and point
|
||||
`ytdlp_pot_provider_url` at it.
|
||||
|
||||
`ytdlp_clients` retries the download against each listed player client in
|
||||
turn, so a client that YouTube has broken is skipped automatically.
|
||||
`ytdlp_cookies` (a Netscape cookie jar, for age-gated videos) and
|
||||
`ytdlp_impersonate` (a browser TLS fingerprint, e.g. `"chrome"`) are the other
|
||||
two levers. Do not use a cookie jar from an account you care about — YouTube
|
||||
bans for automated access.
|
||||
|
||||
Failed downloads leave the work dir intact, so re-running the job resumes
|
||||
rather than starting over.
|
||||
|
||||
## Regenerating the web UI screenshot
|
||||
|
||||
`scripts/screenshot_webui.py` seeds the real app with a fixed set of jobs
|
||||
covering every status and screenshots it to `docs/images/webui.png`. It uses
|
||||
`ffmpeg` to give the selected song a playable folder so the shot includes the
|
||||
player; without `ffmpeg` on `PATH` it still runs, just without it.
|
||||
|
||||
```sh
|
||||
pip install -e '.[dev]'
|
||||
playwright install chromium
|
||||
python scripts/screenshot_webui.py
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
[AGPLv3](LICENSE)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
whisper_url = "http://whisper.intranet:9000/v1"
|
||||
# whisper_api_key = ""
|
||||
# whisper_model = "whisper-1"
|
||||
# whisper_api = "openai" # or "asr-webservice" for onerahmet/openai-whisper-asr-webservice
|
||||
# (set whisper_url without the /v1 suffix, e.g. "http://whisper.intranet:9000")
|
||||
output_dir = "/data/output"
|
||||
work_dir = "/data/work"
|
||||
creator = "brad"
|
||||
@@ -9,3 +11,13 @@ creator = "brad"
|
||||
# device = "cuda"
|
||||
bind_host = "0.0.0.0"
|
||||
# port = 8000
|
||||
|
||||
# YouTube downloads. Each player client is tried in order until one works;
|
||||
# "default" means whatever yt-dlp currently picks for itself.
|
||||
# ytdlp_clients = ["default", "tv"]
|
||||
# ytdlp_cookies = "/config/youtube-cookies.txt" # Netscape cookie jar
|
||||
# ytdlp_impersonate = "chrome" # TLS fingerprint to present
|
||||
# ytdlp_pot_provider_url = "http://bgutil-provider:4416"
|
||||
# Newer yt-dlp releases are fetched from PyPI at runtime and tried before the
|
||||
# bundled one; "bundled" turns that off, a release number pins one.
|
||||
# ytdlp_version = "auto"
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
services:
|
||||
karaoke-maker:
|
||||
songtrace:
|
||||
build: .
|
||||
ports:
|
||||
- "8000:8000"
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 84 KiB |
+11
-5
@@ -1,10 +1,16 @@
|
||||
[project]
|
||||
name = "karaoke-maker"
|
||||
name = "songtrace"
|
||||
version = "0.1.0"
|
||||
description = "Generate UltraStar/Melody Mania karaoke songs from a video and reference lyrics"
|
||||
requires-python = ">=3.11"
|
||||
license = "AGPL-3.0-or-later"
|
||||
dependencies = [
|
||||
"yt-dlp",
|
||||
# yt-dlp declares no runtime deps of its own; the extras are what make
|
||||
# YouTube work. "default" pulls yt-dlp-ejs (JS challenge solving) and the
|
||||
# requests/urllib3 handlers, "deno" ships the JS runtime that executes it,
|
||||
# and "curl-cffi" provides the --impersonate targets. Without them YouTube
|
||||
# extraction degrades to HTTP 403 on the media URLs.
|
||||
"yt-dlp[default,curl-cffi,deno]",
|
||||
"demucs",
|
||||
"torch",
|
||||
"torchaudio",
|
||||
@@ -22,10 +28,10 @@ dependencies = [
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest"]
|
||||
dev = ["pytest", "playwright"]
|
||||
|
||||
[project.scripts]
|
||||
karaoke-maker = "karaoke_maker.cli:main"
|
||||
songtrace = "songtrace.cli:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
@@ -35,7 +41,7 @@ build-backend = "setuptools.build_meta"
|
||||
where = ["src"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
karaoke_maker = ["static/*", "static/fonts/*"]
|
||||
songtrace = ["static/*", "static/fonts/*"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
"""Regenerate docs/images/webui.png against the current web UI.
|
||||
|
||||
Seeds the real app (via create_app's fake-runner seam, same as tests/test_web.py)
|
||||
with a fixed set of jobs covering every status, then screenshots it.
|
||||
|
||||
One-time setup: pip install -e '.[dev]' && playwright install chromium
|
||||
Run: python scripts/screenshot_webui.py
|
||||
"""
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import uvicorn
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import PipelineError, PipelineResult, song_slug
|
||||
from songtrace.web import create_app
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
OUT_IMAGE = REPO_ROOT / "docs" / "images" / "webui.png"
|
||||
STAGES = ["acquire", "separate", "transcribe", "align", "pitch", "notes", "grid", "package"]
|
||||
DEFAULT_LYRICS = "[verse]\n...\n[chorus]\n...\n"
|
||||
|
||||
# public domain (16th century English folk song, author unknown)
|
||||
GREENSLEEVES_LYRICS = """\
|
||||
Alas, my love, you do me wrong
|
||||
To cast me off discourteously
|
||||
For I have loved you well and long
|
||||
Delighting in your company
|
||||
|
||||
Greensleeves was all my joy
|
||||
Greensleeves was my delight
|
||||
Greensleeves was my heart of gold
|
||||
And who but my lady Greensleeves
|
||||
|
||||
I have been ready at your hand
|
||||
To grant whatever you would crave
|
||||
I have both waged life and land
|
||||
Your love and good will for to have
|
||||
|
||||
Greensleeves was all my joy
|
||||
Greensleeves was my delight
|
||||
Greensleeves was my heart of gold
|
||||
And who but my lady Greensleeves
|
||||
"""
|
||||
|
||||
# order matters: the app has one worker thread, so each job runs in submission
|
||||
# order. "running" must be submitted after the done/failed ones complete and
|
||||
# before "queued", so it's the one still in progress when "queued" arrives.
|
||||
SONGS = [
|
||||
("Every Single Night", "Fiona Apple", "done",
|
||||
{"warnings": ["1 low-confidence note near 2:41–2:44"]}),
|
||||
("This Year", "The Mountain Goats", "done", {"warnings": []}),
|
||||
("Pink Moon", "Nick Drake", "done",
|
||||
{"warnings": ["2 low-confidence notes near 0:38–0:41",
|
||||
"1 low-confidence note near 1:22–1:24"]}),
|
||||
("Landslide", "Fleetwood Mac", "done",
|
||||
{"warnings": ["1 low-confidence note near 0:52–0:55"]}),
|
||||
("Greensleeves", "Traditional", "done",
|
||||
{"lyrics": GREENSLEEVES_LYRICS,
|
||||
"warnings": ["1 low-confidence note near 0:45–0:47",
|
||||
"1 line dropped (not sung) near \"for i have loved\"",
|
||||
"missing lyrics near 2:10–2:18"],
|
||||
"review": {
|
||||
"lines": [
|
||||
# Pitches are the real Greensleeves melody (E natural minor, as
|
||||
# arranged by Gilbert DeBenedetti, "Level 4 - Flowing"), converted
|
||||
# to UltraStar semitones (0 = C4 = MIDI 60): E5=16 F#5=18 G5=19
|
||||
# A5=21 B4=11 D5=14.
|
||||
{"index": 0, "text": "Alas, my love, you do me wrong",
|
||||
"dropped": False,
|
||||
"notes": [{"start": 10.0, "end": 10.4, "pitch": 16, "low_confidence": False},
|
||||
{"start": 10.4, "end": 10.6, "pitch": 18, "low_confidence": False},
|
||||
{"start": 10.6, "end": 11.2, "pitch": 19, "low_confidence": False},
|
||||
{"start": 11.2, "end": 11.5, "pitch": 21, "low_confidence": False},
|
||||
{"start": 11.5, "end": 11.7, "pitch": 19, "low_confidence": False},
|
||||
{"start": 11.7, "end": 11.9, "pitch": 18, "low_confidence": False},
|
||||
{"start": 11.9, "end": 12.9, "pitch": 14, "low_confidence": False}],
|
||||
"words": [{"text": "Alas,", "start": 10.0, "end": 10.4},
|
||||
{"text": "my", "start": 10.4, "end": 10.6},
|
||||
{"text": "love,", "start": 10.6, "end": 11.2},
|
||||
{"text": "you", "start": 11.2, "end": 11.5},
|
||||
{"text": "do", "start": 11.5, "end": 11.7},
|
||||
{"text": "me", "start": 11.7, "end": 11.9},
|
||||
{"text": "wrong", "start": 11.9, "end": 12.9}]},
|
||||
{"index": 1, "text": "To cast me off discourteously",
|
||||
"dropped": False,
|
||||
"notes": [{"start": 44.0, "end": 44.3, "pitch": 11, "low_confidence": False},
|
||||
{"start": 44.3, "end": 44.6, "pitch": 16, "low_confidence": False},
|
||||
{"start": 44.6, "end": 44.8, "pitch": 18, "low_confidence": False},
|
||||
{"start": 44.8, "end": 45.0, "pitch": 19, "low_confidence": False},
|
||||
{"start": 45.0, "end": 45.4, "pitch": 18, "low_confidence": True}],
|
||||
"words": [{"text": "To", "start": 44.0, "end": 44.3},
|
||||
{"text": "cast", "start": 44.3, "end": 44.6},
|
||||
{"text": "me", "start": 44.6, "end": 44.8},
|
||||
{"text": "off", "start": 44.8, "end": 45.0},
|
||||
{"text": "discourteously", "start": 45.0, "end": 45.4}]},
|
||||
{"index": 2, "text": "For I have loved you well and long",
|
||||
"dropped": True, "notes": [], "words": []},
|
||||
],
|
||||
"missing_spans": [{"start": 130.0, "end": 138.0}],
|
||||
}}),
|
||||
("Such Great Heights", "The Postal Service", "failed",
|
||||
{"error": "yt-dlp: unable to download video data: HTTP Error 403: Forbidden"}),
|
||||
("Time Escaping", "Big Thief", "running", {}),
|
||||
("Vampire in the Corner", "Magdalena Bay", "queued", {}),
|
||||
]
|
||||
SELECTED = "Greensleeves"
|
||||
REVIEW_HEADING = "REVIEW"
|
||||
|
||||
running_event = threading.Event()
|
||||
|
||||
|
||||
def make_runner():
|
||||
specs = {title: (kind, extra) for title, _, kind, extra in SONGS}
|
||||
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
kind, extra = specs[inp.title]
|
||||
if kind == "done":
|
||||
for s in STAGES:
|
||||
log(f"[{s}] running")
|
||||
if "review" in extra:
|
||||
work = cfg.work_dir / song_slug(inp)
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
notes = [
|
||||
{**n, "duration": n["end"] - n["start"], "text": "~",
|
||||
"line_index": line["index"]}
|
||||
for line in extra["review"]["lines"] for n in line["notes"]
|
||||
]
|
||||
(work / "notes_sec.json").write_text(json.dumps({
|
||||
"notes": notes, "breaks": [], "warnings": extra["warnings"]}))
|
||||
(work / "align_meta.json").write_text(json.dumps({
|
||||
"dropped_lines": [line["index"] for line in extra["review"]["lines"]
|
||||
if line["dropped"]],
|
||||
"missing_spans": [[s["start"], s["end"]]
|
||||
for s in extra["review"]["missing_spans"]],
|
||||
}))
|
||||
(work / "alignment.json").write_text(json.dumps([
|
||||
{"word": w["text"], "start": w["start"], "end": w["end"],
|
||||
"score": 0.9, "line_index": line["index"]}
|
||||
for line in extra["review"]["lines"] for w in line.get("words", [])
|
||||
]))
|
||||
out = cfg.output_dir / f"{inp.artist} - {inp.title}"
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
return PipelineResult(output_dir=out, warnings=extra["warnings"])
|
||||
if kind == "failed":
|
||||
log("[acquire] running")
|
||||
raise PipelineError(extra["error"])
|
||||
for s in ["acquire", "separate", "transcribe", "align"]:
|
||||
log(f"[{s}] running")
|
||||
running_event.wait()
|
||||
return PipelineResult(output_dir=cfg.output_dir / "x", warnings=[])
|
||||
|
||||
return runner
|
||||
|
||||
|
||||
def start_server(tmp_path: Path) -> tuple[str, Config]:
|
||||
cfg = Config(whisper_url="http://localhost:9000/v1", output_dir=tmp_path / "out",
|
||||
work_dir=tmp_path / "work", bind_host="127.0.0.1", port=8811)
|
||||
app = create_app(cfg, runner=make_runner())
|
||||
server = uvicorn.Server(uvicorn.Config(app, host=cfg.bind_host, port=cfg.port,
|
||||
log_level="warning"))
|
||||
threading.Thread(target=server.run, daemon=True).start()
|
||||
return f"http://{cfg.bind_host}:{cfg.port}", cfg
|
||||
|
||||
|
||||
def install_media(cfg: Config) -> bool:
|
||||
"""Give the selected song a playable folder, so the shot shows the player.
|
||||
|
||||
A flat fill rather than a test pattern: the point is the transport and the
|
||||
lyric tracking underneath it, not the picture. Skipped without ffmpeg.
|
||||
"""
|
||||
if shutil.which("ffmpeg") is None:
|
||||
print("ffmpeg not found; screenshotting without the player")
|
||||
return False
|
||||
artist, title = next((a, t) for t, a, _, _ in SONGS if t == SELECTED)
|
||||
slug = f"{artist} - {title}"
|
||||
out = cfg.output_dir / slug
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-loglevel", "error", "-f", "lavfi",
|
||||
"-i", "color=c=0x1A1430:s=640x360:d=60", "-c:v", "libx264",
|
||||
"-pix_fmt", "yuv420p", "-preset", "ultrafast", "-an",
|
||||
str(out / f"{slug}.mp4")], check=True)
|
||||
subprocess.run(
|
||||
["ffmpeg", "-y", "-loglevel", "error", "-f", "lavfi",
|
||||
"-i", "anullsrc=r=44100:cl=stereo:d=60", "-b:a", "64k",
|
||||
str(out / f"{slug}.mp3")], check=True)
|
||||
(out / f"{slug}.txt").write_text(
|
||||
f"#TITLE:{title}\n#ARTIST:{artist}\n#MP3:{slug}.mp3\n#VIDEO:{slug}.mp4\n"
|
||||
f"#BPM:240\n#GAP:10000\n: 0 4 16 Alas,\nE\n", encoding="utf-8")
|
||||
return True
|
||||
|
||||
|
||||
def seed_jobs(base_url: str) -> None:
|
||||
with httpx.Client(timeout=10) as client:
|
||||
for title, artist, kind, extra in SONGS:
|
||||
resp = client.post(f"{base_url}/api/jobs", data={
|
||||
"title": title, "artist": artist,
|
||||
"lyrics": extra.get("lyrics", DEFAULT_LYRICS),
|
||||
"youtube_url": "https://youtu.be/00000000000"})
|
||||
resp.raise_for_status()
|
||||
job_id = resp.json()["id"]
|
||||
if kind == "queued":
|
||||
continue
|
||||
for _ in range(200):
|
||||
job = client.get(f"{base_url}/api/jobs/{job_id}").json()
|
||||
if kind == "running" and job["stage"] == "align":
|
||||
break
|
||||
if job["status"] in ("done", "failed"):
|
||||
break
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def capture(base_url: str, with_player: bool) -> None:
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 1000})
|
||||
page.goto(base_url, wait_until="networkidle")
|
||||
page.get_by_text(SELECTED, exact=True).click()
|
||||
page.wait_for_timeout(500)
|
||||
if with_player:
|
||||
# paused mid-line, so the shot carries the playhead and the sung /
|
||||
# current word colouring rather than an untouched transport
|
||||
page.evaluate("document.getElementById('player-media').currentTime = 11.6")
|
||||
page.wait_for_timeout(400)
|
||||
# scrolled to the review: the player pins to the top of #main and
|
||||
# the tracked lines sit under it, which is the panel in use
|
||||
page.evaluate("""() => {
|
||||
const main = document.getElementById('main');
|
||||
main.scrollTo(0, document.getElementById('review').offsetTop - 12);
|
||||
}""")
|
||||
page.wait_for_timeout(400)
|
||||
OUT_IMAGE.parent.mkdir(parents=True, exist_ok=True)
|
||||
page.screenshot(path=str(OUT_IMAGE))
|
||||
browser.close()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
base_url, cfg = start_server(Path(tmp))
|
||||
time.sleep(1)
|
||||
seed_jobs(base_url)
|
||||
with_player = install_media(cfg)
|
||||
capture(base_url, with_player)
|
||||
print(f"wrote {OUT_IMAGE}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,36 +0,0 @@
|
||||
import os
|
||||
import tomllib
|
||||
from dataclasses import dataclass, fields
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
whisper_url: str
|
||||
whisper_api_key: str | None = None
|
||||
whisper_model: str = "whisper-1"
|
||||
output_dir: Path = Path("output")
|
||||
work_dir: Path = Path("work")
|
||||
creator: str = "karaoke-maker"
|
||||
demucs_model: str = "htdemucs_ft"
|
||||
crepe_model: str = "full"
|
||||
device: str = "cuda"
|
||||
bind_host: str = "127.0.0.1"
|
||||
port: int = 8000
|
||||
|
||||
|
||||
_PATH_FIELDS = {"output_dir", "work_dir"}
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> Config:
|
||||
if path is None:
|
||||
env = os.environ.get("KARAOKE_MAKER_CONFIG")
|
||||
path = Path(env) if env else Path.home() / ".config/karaoke-maker/config.toml"
|
||||
data = tomllib.loads(Path(path).read_text())
|
||||
known = {f.name for f in fields(Config)}
|
||||
unknown = set(data) - known
|
||||
if unknown:
|
||||
raise ValueError(f"unknown config keys: {', '.join(sorted(unknown))}")
|
||||
for key in _PATH_FIELDS & set(data):
|
||||
data[key] = Path(data[key])
|
||||
return Config(**data)
|
||||
@@ -1,54 +0,0 @@
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from ..media import ffmpeg
|
||||
from ..pipeline import Ctx, PipelineError, Stage
|
||||
|
||||
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".avif", ".gif"}
|
||||
|
||||
|
||||
def _classify_downloads(dest: Path) -> tuple[Path, Path | None]:
|
||||
files = list(dest.glob("source_video.*"))
|
||||
thumbs = [p for p in files if p.suffix.lower() in IMAGE_EXTENSIONS]
|
||||
videos = [p for p in files if p.suffix.lower() not in IMAGE_EXTENSIONS]
|
||||
if not videos:
|
||||
raise PipelineError("yt-dlp produced no video file")
|
||||
video = max(videos, key=lambda p: p.stat().st_size)
|
||||
return video, thumbs[0] if thumbs else None
|
||||
|
||||
|
||||
def download_youtube(url: str, dest: Path) -> tuple[Path, Path | None]:
|
||||
import yt_dlp
|
||||
opts = {
|
||||
"outtmpl": str(dest / "source_video.%(ext)s"),
|
||||
"format": "bestvideo*+bestaudio/best",
|
||||
"writethumbnail": True,
|
||||
"noplaylist": True,
|
||||
}
|
||||
with yt_dlp.YoutubeDL(opts) as ydl:
|
||||
ydl.extract_info(url, download=True)
|
||||
return _classify_downloads(dest)
|
||||
|
||||
|
||||
def _run(ctx: Ctx) -> None:
|
||||
if bool(ctx.inp.youtube_url) == bool(ctx.inp.video_path):
|
||||
raise PipelineError("provide exactly one of youtube_url / video_path")
|
||||
thumb = None
|
||||
if ctx.inp.youtube_url:
|
||||
video, thumb = download_youtube(ctx.inp.youtube_url, ctx.work)
|
||||
else:
|
||||
video = ctx.work / f"source_video{ctx.inp.video_path.suffix}"
|
||||
shutil.copyfile(ctx.inp.video_path, video)
|
||||
ffmpeg("-i", str(video), "-vn", "-ac", "2", "-ar", "44100",
|
||||
str(ctx.work / "audio.wav"))
|
||||
meta = {"video_file": video.name, "thumbnail": thumb.name if thumb else None}
|
||||
(ctx.work / "meta.json").write_text(json.dumps(meta))
|
||||
|
||||
|
||||
def acquire_stage() -> Stage:
|
||||
return Stage(
|
||||
name="acquire",
|
||||
outputs=lambda ctx: [ctx.work / "meta.json", ctx.work / "audio.wav"],
|
||||
run=_run,
|
||||
)
|
||||
@@ -1,43 +0,0 @@
|
||||
import difflib
|
||||
import json
|
||||
|
||||
from ..lyrics import RefLyrics, alignment_token
|
||||
from ..pipeline import Ctx, Stage
|
||||
from ..whisper_client import TranscriptSegment, transcribe
|
||||
|
||||
|
||||
def vocal_sections(segments, min_gap: float = 6.0, pad: float = 1.0):
|
||||
sections = []
|
||||
for s in sorted(segments, key=lambda s: s.start):
|
||||
if sections and s.start - sections[-1][1] < min_gap:
|
||||
sections[-1][1] = max(sections[-1][1], s.end)
|
||||
else:
|
||||
sections.append([s.start, s.end])
|
||||
return [(max(0.0, a - pad), b + pad) for a, b in sections]
|
||||
|
||||
|
||||
def _tokens(text: str) -> list[str]:
|
||||
return [t for w in text.split() if (t := alignment_token(w))]
|
||||
|
||||
|
||||
def lyrics_similarity(segments, ref: RefLyrics) -> float:
|
||||
heard = [t for s in segments for t in _tokens(s.text)]
|
||||
expected = [t for w in ref.words if (t := alignment_token(w))]
|
||||
return difflib.SequenceMatcher(a=heard, b=expected).ratio()
|
||||
|
||||
|
||||
def _run(ctx: Ctx) -> None:
|
||||
segments = transcribe(ctx.cfg, ctx.work / "vocals.wav", ctx.inp.language)
|
||||
(ctx.work / "transcript.json").write_text(json.dumps(
|
||||
[{"start": s.start, "end": s.end, "text": s.text} for s in segments]))
|
||||
|
||||
|
||||
def transcribe_stage() -> Stage:
|
||||
return Stage(name="transcribe",
|
||||
outputs=lambda ctx: [ctx.work / "transcript.json"],
|
||||
run=_run)
|
||||
|
||||
|
||||
def load_transcript(ctx: Ctx) -> list[TranscriptSegment]:
|
||||
data = json.loads((ctx.work / "transcript.json").read_text())
|
||||
return [TranscriptSegment(**d) for d in data]
|
||||
@@ -1,39 +0,0 @@
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from .config import Config
|
||||
from .languages import LANGUAGES
|
||||
|
||||
|
||||
@dataclass
|
||||
class TranscriptSegment:
|
||||
start: float
|
||||
end: float
|
||||
text: str
|
||||
|
||||
|
||||
def transcribe(cfg: Config, audio_path: Path, language: str | None = None,
|
||||
client: httpx.Client | None = None) -> list[TranscriptSegment]:
|
||||
headers = {}
|
||||
if cfg.whisper_api_key:
|
||||
headers["Authorization"] = f"Bearer {cfg.whisper_api_key}"
|
||||
data = {"model": cfg.whisper_model, "response_format": "verbose_json"}
|
||||
if language:
|
||||
entry = LANGUAGES.get(language.lower())
|
||||
data["language"] = entry.iso if entry else language
|
||||
own = client is None
|
||||
client = client or httpx.Client(timeout=600)
|
||||
try:
|
||||
with audio_path.open("rb") as f:
|
||||
resp = client.post(
|
||||
f"{cfg.whisper_url.rstrip('/')}/audio/transcriptions",
|
||||
headers=headers, data=data,
|
||||
files={"file": (audio_path.name, f, "audio/wav")})
|
||||
resp.raise_for_status()
|
||||
return [TranscriptSegment(s["start"], s["end"], s["text"].strip())
|
||||
for s in resp.json().get("segments", [])]
|
||||
finally:
|
||||
if own:
|
||||
client.close()
|
||||
@@ -0,0 +1,14 @@
|
||||
from pathlib import Path
|
||||
|
||||
import soundfile as sf
|
||||
import torch
|
||||
|
||||
|
||||
def load_wav(path: Path) -> tuple[torch.Tensor, int]:
|
||||
"""Load a WAV file as a (channels, samples) float32 tensor.
|
||||
|
||||
Reads via soundfile rather than torchaudio.load, whose default backend
|
||||
now requires the separate torchcodec package.
|
||||
"""
|
||||
data, sr = sf.read(str(path), dtype="float32", always_2d=True)
|
||||
return torch.from_numpy(data.T).contiguous(), sr
|
||||
@@ -16,7 +16,7 @@ def _positive_float(value: str) -> float:
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(prog="karaoke-maker")
|
||||
parser = argparse.ArgumentParser(prog="songtrace")
|
||||
parser.add_argument("--version", action="store_true")
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
|
||||
@@ -72,7 +72,7 @@ def _cmd_make(args) -> int:
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
if args.version:
|
||||
print(f"karaoke-maker {version('karaoke-maker')}")
|
||||
print(f"songtrace {version('songtrace')}")
|
||||
return 0
|
||||
if args.command == "make":
|
||||
return _cmd_make(args)
|
||||
@@ -0,0 +1,46 @@
|
||||
import os
|
||||
import tomllib
|
||||
from dataclasses import dataclass, field, fields
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
whisper_url: str
|
||||
whisper_api_key: str | None = None
|
||||
whisper_model: str = "whisper-1"
|
||||
whisper_api: str = "openai"
|
||||
output_dir: Path = Path("output")
|
||||
work_dir: Path = Path("work")
|
||||
creator: str = "songtrace"
|
||||
demucs_model: str = "htdemucs_ft"
|
||||
crepe_model: str = "full"
|
||||
device: str = "cuda"
|
||||
bind_host: str = "127.0.0.1"
|
||||
port: int = 8000
|
||||
# YouTube player clients to try, in order, until one yields a download.
|
||||
# "default" means yt-dlp's own choice; see its extractor-args docs.
|
||||
ytdlp_clients: list[str] = field(default_factory=lambda: ["default", "tv"])
|
||||
ytdlp_cookies: Path | None = None
|
||||
ytdlp_impersonate: str | None = None
|
||||
ytdlp_pot_provider_url: str | None = None
|
||||
# "auto" keeps yt-dlp current from PyPI, "bundled" uses only the copy
|
||||
# installed with songtrace, and a release such as "2026.08.19" pins that.
|
||||
ytdlp_version: str = "auto"
|
||||
|
||||
|
||||
_PATH_FIELDS = {"output_dir", "work_dir", "ytdlp_cookies"}
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> Config:
|
||||
if path is None:
|
||||
env = os.environ.get("SONGTRACE_CONFIG")
|
||||
path = Path(env) if env else Path.home() / ".config/songtrace/config.toml"
|
||||
data = tomllib.loads(Path(path).read_text())
|
||||
known = {f.name for f in fields(Config)}
|
||||
unknown = set(data) - known
|
||||
if unknown:
|
||||
raise ValueError(f"unknown config keys: {', '.join(sorted(unknown))}")
|
||||
for key in _PATH_FIELDS & set(data):
|
||||
data[key] = Path(data[key])
|
||||
return Config(**data)
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Handing the GPU back between stages.
|
||||
|
||||
The GPU this runs on is shared, and `songtrace serve` outlives every job it
|
||||
runs, so a stage that caches its weights holds VRAM until the process exits —
|
||||
including while a later stage is waiting on the Whisper server, which can be
|
||||
minutes of a GPU nothing is using. Two things have to be undone: the caches
|
||||
themselves, and PyTorch's allocator, which keeps freed blocks for reuse rather
|
||||
than returning them to the driver.
|
||||
|
||||
A stage that caches a model registers how to drop it with @on_release, so this
|
||||
module needs to know nothing about the pipeline. Registration happens at import
|
||||
time, which run_pipeline has already done by the time it calls release().
|
||||
"""
|
||||
import gc
|
||||
import sys
|
||||
from typing import Callable
|
||||
|
||||
_callbacks: list[Callable[[], None]] = []
|
||||
|
||||
|
||||
def on_release(fn: Callable[[], None]) -> Callable[[], None]:
|
||||
"""Register a callback that drops a cached model."""
|
||||
_callbacks.append(fn)
|
||||
return fn
|
||||
|
||||
|
||||
def release() -> int:
|
||||
"""Drop every cached model and return its VRAM to the driver.
|
||||
|
||||
Answers the bytes handed back, which is the only part of this a caller can
|
||||
show someone. Safe whenever: a stage that never loaded anything has nothing
|
||||
to drop, and the allocator is left alone unless this process actually
|
||||
initialised CUDA.
|
||||
"""
|
||||
for callback in _callbacks:
|
||||
callback()
|
||||
# weights reach the allocator's free list only once the last reference
|
||||
# goes, and a module graph is cyclic enough to outlive refcounting
|
||||
gc.collect()
|
||||
torch = sys.modules.get("torch")
|
||||
if torch is None or not torch.cuda.is_available():
|
||||
return 0
|
||||
if not torch.cuda.is_initialized():
|
||||
return 0
|
||||
before = torch.cuda.memory_reserved()
|
||||
torch.cuda.empty_cache()
|
||||
return max(before - torch.cuda.memory_reserved(), 0)
|
||||
@@ -23,6 +23,7 @@ class TimedNote:
|
||||
pitch: int # UltraStar semitone (0 = C4 = MIDI 60)
|
||||
text: str # display text incl. leading space / "~"
|
||||
low_confidence: bool
|
||||
line_index: int
|
||||
|
||||
|
||||
def hz_to_midi_float(hz: float) -> float:
|
||||
@@ -78,7 +79,8 @@ def syllabify(word: str, language: str | None) -> list[str]:
|
||||
return parts or [word]
|
||||
|
||||
|
||||
def build_notes(rows: list[dict], times, f0, conf, language: str | None
|
||||
def build_notes(rows: list[dict], times, f0, conf, language: str | None,
|
||||
dropped_lines=(), missing_spans=(), ref_lines=None
|
||||
) -> tuple[list[TimedNote], list[float], list[str]]:
|
||||
notes: list[TimedNote] = []
|
||||
breaks: list[float] = []
|
||||
@@ -92,7 +94,8 @@ def build_notes(rows: list[dict], times, f0, conf, language: str | None
|
||||
if not segs:
|
||||
pitch = notes[-1].pitch if notes else 0
|
||||
notes.append(_note(row["start"], row["end"] - row["start"], pitch,
|
||||
_text(row["word"], 0, line_initial), True))
|
||||
_text(row["word"], 0, line_initial), True,
|
||||
row["line_index"]))
|
||||
continue
|
||||
low = row["score"] < LOW_SCORE
|
||||
sylls = syllabify(row["word"], language)
|
||||
@@ -100,14 +103,16 @@ def build_notes(rows: list[dict], times, f0, conf, language: str | None
|
||||
segs = _subdivide(segs, len(sylls))
|
||||
for i, (s, e, midi) in enumerate(segs):
|
||||
text = _text(sylls[i], i, line_initial) if i < len(sylls) else "~"
|
||||
notes.append(_note(s, e - s, round(midi) - 60, text, low))
|
||||
notes.append(_note(s, e - s, round(midi) - 60, text, low,
|
||||
row["line_index"]))
|
||||
_deoverlap(notes)
|
||||
return notes, breaks, _warnings(notes)
|
||||
return notes, breaks, _warnings(notes, dropped_lines, missing_spans, ref_lines or [])
|
||||
|
||||
|
||||
def _note(start: float, duration: float, pitch: int, text: str,
|
||||
low: bool) -> TimedNote:
|
||||
return TimedNote(float(start), float(duration), int(pitch), text, low)
|
||||
low: bool, line_index: int) -> TimedNote:
|
||||
return TimedNote(float(start), float(duration), int(pitch), text, low,
|
||||
int(line_index))
|
||||
|
||||
|
||||
def _deoverlap(notes: list[TimedNote]) -> None:
|
||||
@@ -153,7 +158,8 @@ def _fmt(t: float) -> str:
|
||||
return f"{int(t // 60)}:{int(t % 60):02d}"
|
||||
|
||||
|
||||
def _warnings(notes: list[TimedNote]) -> list[str]:
|
||||
def _warnings(notes: list[TimedNote], dropped_lines=(), missing_spans=(),
|
||||
ref_lines=()) -> list[str]:
|
||||
warnings, run = [], []
|
||||
for n in list(notes) + [None]:
|
||||
if n is not None and n.low_confidence:
|
||||
@@ -164,4 +170,21 @@ def _warnings(notes: list[TimedNote]) -> list[str]:
|
||||
f"{len(run)} low-confidence note{'' if len(run) == 1 else 's'} "
|
||||
f"near {_fmt(first.start)}–{_fmt(last.start + last.duration)}")
|
||||
run = []
|
||||
for group in _consecutive_runs(dropped_lines):
|
||||
n = len(group)
|
||||
text = " ".join(ref_lines[group[0]]) if group[0] < len(ref_lines) else ""
|
||||
warnings.append(
|
||||
f"{n} line{'' if n == 1 else 's'} dropped (not sung) near \"{text}\"")
|
||||
for a, b in missing_spans:
|
||||
warnings.append(f"missing lyrics near {_fmt(a)}–{_fmt(b)}")
|
||||
return warnings
|
||||
|
||||
|
||||
def _consecutive_runs(indices) -> list[list[int]]:
|
||||
runs = []
|
||||
for i in sorted(indices):
|
||||
if runs and i == runs[-1][-1] + 1:
|
||||
runs[-1].append(i)
|
||||
else:
|
||||
runs.append([i])
|
||||
return runs
|
||||
@@ -3,6 +3,7 @@ from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
from . import models
|
||||
from .config import Config
|
||||
|
||||
class PipelineError(Exception):
|
||||
@@ -45,6 +46,12 @@ def song_slug(inp: SongInput) -> str:
|
||||
name = f"{inp.artist} - {inp.title}"
|
||||
return name.replace("/", "").replace("\0", "")
|
||||
|
||||
def read_warnings(work: Path) -> list[str]:
|
||||
notes_file = work / "notes_sec.json"
|
||||
if not notes_file.exists():
|
||||
return []
|
||||
return json.loads(notes_file.read_text()).get("warnings", [])
|
||||
|
||||
def _real_stages() -> list[Stage]:
|
||||
from . import stages
|
||||
return stages.all_stages()
|
||||
@@ -68,9 +75,14 @@ def run_pipeline(inp: SongInput, cfg: Config, log=print,
|
||||
log(f"[{stage.name}] cached, skipping")
|
||||
continue
|
||||
log(f"[{stage.name}] running")
|
||||
stage.run(ctx)
|
||||
warnings = []
|
||||
notes_file = work / "notes_sec.json"
|
||||
if notes_file.exists():
|
||||
warnings = json.loads(notes_file.read_text()).get("warnings", [])
|
||||
return PipelineResult(output_dir=cfg.output_dir / song_slug(inp), warnings=warnings)
|
||||
# per stage rather than per job: each model is used by one stage, so
|
||||
# holding it past that only keeps VRAM from whatever else wants it
|
||||
try:
|
||||
stage.run(ctx)
|
||||
finally:
|
||||
# logged only when it actually gave something back, so the line is
|
||||
# evidence rather than noise on a box where this matters
|
||||
if freed := models.release():
|
||||
log(f"[{stage.name}] released {freed / 2**20:.0f} MiB of GPU memory")
|
||||
return PipelineResult(output_dir=cfg.output_dir / song_slug(inp),
|
||||
warnings=read_warnings(work))
|
||||
@@ -38,6 +38,17 @@ def plan(old: SongInput, new: SongInput) -> str | None:
|
||||
return min((FIELD_STAGE[f] for f in changed), key=STAGE_ORDER.index)
|
||||
|
||||
|
||||
def earliest(*stages: str | None) -> str | None:
|
||||
"""The earliest of the stages given, ignoring None ("nothing forced").
|
||||
|
||||
An operator's explicit choice is combined with plan()'s rather than
|
||||
replacing it: the planner's stage is the latest that is still correct for
|
||||
the edit, so a choice may only ever redo more.
|
||||
"""
|
||||
named = [s for s in stages if s]
|
||||
return min(named, key=STAGE_ORDER.index) if named else None
|
||||
|
||||
|
||||
def relocate_work(old: SongInput, new: SongInput, cfg: Config) -> bool:
|
||||
"""Move the cached work dir when title/artist changed its slug.
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from .. import ytdlp
|
||||
from ..config import Config
|
||||
from ..media import ffmpeg
|
||||
from ..pipeline import Ctx, PipelineError, Stage
|
||||
|
||||
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".avif", ".gif"}
|
||||
# yt-dlp's in-progress scratch files, which a failed attempt leaves behind.
|
||||
PARTIAL_EXTENSIONS = {".part", ".ytdl", ".temp"}
|
||||
|
||||
|
||||
def _classify_downloads(dest: Path) -> tuple[Path, Path | None]:
|
||||
files = [p for p in dest.glob("source_video.*")
|
||||
if p.suffix.lower() not in PARTIAL_EXTENSIONS]
|
||||
thumbs = [p for p in files if p.suffix.lower() in IMAGE_EXTENSIONS]
|
||||
videos = [p for p in files if p.suffix.lower() not in IMAGE_EXTENSIONS]
|
||||
if not videos:
|
||||
raise PipelineError("yt-dlp produced no video file")
|
||||
video = max(videos, key=lambda p: p.stat().st_size)
|
||||
return video, thumbs[0] if thumbs else None
|
||||
|
||||
|
||||
def _ydl_args(cfg: Config, dest: Path, client: str, url: str) -> list[str]:
|
||||
args = [
|
||||
"--output", str(dest / "source_video.%(ext)s"),
|
||||
"--format", "bestvideo*+bestaudio/best",
|
||||
"--write-thumbnail", "--no-playlist", "--no-progress",
|
||||
"--retries", "10", "--fragment-retries", "10",
|
||||
"--extractor-retries", "5",
|
||||
]
|
||||
if client != "default":
|
||||
args += ["--extractor-args", f"youtube:player_client={client}"]
|
||||
if cfg.ytdlp_pot_provider_url:
|
||||
args += ["--extractor-args",
|
||||
f"youtubepot-bgutilhttp:base_url={cfg.ytdlp_pot_provider_url}"]
|
||||
if cfg.ytdlp_cookies:
|
||||
args += ["--cookies", str(cfg.ytdlp_cookies)]
|
||||
if cfg.ytdlp_impersonate:
|
||||
args += ["--impersonate", cfg.ytdlp_impersonate]
|
||||
return [*args, "--", url]
|
||||
|
||||
|
||||
def download_youtube(url: str, dest: Path, cfg: Config,
|
||||
log=print) -> tuple[Path, Path | None]:
|
||||
clients = list(cfg.ytdlp_clients) or ["default"]
|
||||
tried: list[str] = []
|
||||
last: Exception | None = None
|
||||
# If every copy fails, look for a release newer than the daily check found.
|
||||
for refresh in (False, True):
|
||||
for ydl in ytdlp.candidates(cfg, log, refresh):
|
||||
if ydl.version in tried:
|
||||
continue
|
||||
tried.append(ydl.version)
|
||||
for client in clients:
|
||||
try:
|
||||
ydl.run(_ydl_args(cfg, dest, client, url), log)
|
||||
return _classify_downloads(dest)
|
||||
except PipelineError as e:
|
||||
last = e
|
||||
log(f"yt-dlp {ydl.version} player_client={client} "
|
||||
f"failed: {e}")
|
||||
raise PipelineError(
|
||||
f"yt-dlp {', '.join(tried)} could not download {url} after trying "
|
||||
f"player_client={', '.join(clients)}. Last error: {last}. "
|
||||
"See the yt-dlp options in config.toml (cookies, impersonation, "
|
||||
"PO token provider).")
|
||||
|
||||
|
||||
def _run(ctx: Ctx) -> None:
|
||||
if bool(ctx.inp.youtube_url) == bool(ctx.inp.video_path):
|
||||
raise PipelineError("provide exactly one of youtube_url / video_path")
|
||||
thumb = None
|
||||
if ctx.inp.youtube_url:
|
||||
video, thumb = download_youtube(ctx.inp.youtube_url, ctx.work,
|
||||
ctx.cfg, ctx.log)
|
||||
else:
|
||||
video = ctx.work / f"source_video{ctx.inp.video_path.suffix}"
|
||||
shutil.copyfile(ctx.inp.video_path, video)
|
||||
ffmpeg("-i", str(video), "-vn", "-ac", "2", "-ar", "44100",
|
||||
str(ctx.work / "audio.wav"))
|
||||
meta = {"video_file": video.name, "thumbnail": thumb.name if thumb else None}
|
||||
(ctx.work / "meta.json").write_text(json.dumps(meta))
|
||||
|
||||
|
||||
def acquire_stage() -> Stage:
|
||||
return Stage(
|
||||
name="acquire",
|
||||
outputs=lambda ctx: [ctx.work / "meta.json", ctx.work / "audio.wav"],
|
||||
run=_run,
|
||||
)
|
||||
@@ -1,12 +1,15 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from ..lyrics import RefLyrics, alignment_token, normalize_lyrics
|
||||
from ..lyrics import alignment_token, normalize_lyrics
|
||||
from ..models import on_release
|
||||
from ..pipeline import Ctx, PipelineError, Stage
|
||||
from .transcribe import load_transcript, lyrics_similarity, vocal_sections
|
||||
from .transcribe import classify_lyrics_match, load_transcript, lyrics_similarity, vocal_sections
|
||||
|
||||
SIMILARITY_THRESHOLD = 0.4
|
||||
|
||||
# a stage aligns section by section, so the model is cached across the calls
|
||||
# within one run and dropped at the end of it
|
||||
_model_cache = {}
|
||||
|
||||
|
||||
@@ -17,13 +20,20 @@ def _get_model(device: str):
|
||||
return _model_cache[device]
|
||||
|
||||
|
||||
@on_release
|
||||
def unload_model() -> None:
|
||||
_model_cache.clear()
|
||||
|
||||
|
||||
def force_align_section(vocals_path: Path, start: float, end: float,
|
||||
tokens: list[str], device: str) -> list[tuple[float, float, float]]:
|
||||
import torch
|
||||
import torchaudio
|
||||
from torchaudio.pipelines import MMS_FA as bundle
|
||||
|
||||
waveform, sr = torchaudio.load(str(vocals_path))
|
||||
from ..audio_io import load_wav
|
||||
|
||||
waveform, sr = load_wav(vocals_path)
|
||||
waveform = waveform.mean(0, keepdim=True)
|
||||
lo, hi = int(start * sr), min(int(end * sr), waveform.shape[1])
|
||||
waveform = torchaudio.functional.resample(
|
||||
@@ -40,16 +50,15 @@ def force_align_section(vocals_path: Path, start: float, end: float,
|
||||
sum(t.score for t in s) / len(s)) for s in spans]
|
||||
|
||||
|
||||
def _assign_lines_to_sections(ref: RefLyrics, sections) -> list[list[int]]:
|
||||
"""Split lyric lines across vocal sections proportionally to section duration."""
|
||||
def _assign_lines_to_sections(line_indices: list[int], sections) -> list[list[int]]:
|
||||
"""Split matched lyric line indices across vocal sections proportionally to section duration."""
|
||||
total = sum(b - a for a, b in sections) or 1.0
|
||||
n_lines = len(ref.lines)
|
||||
n = len(line_indices)
|
||||
groups, elapsed, start = [], 0.0, 0
|
||||
for i, (a, b) in enumerate(sections):
|
||||
elapsed += b - a
|
||||
stop = n_lines if i == len(sections) - 1 else min(
|
||||
n_lines, round(n_lines * elapsed / total))
|
||||
groups.append(list(range(start, max(stop, start))))
|
||||
stop = n if i == len(sections) - 1 else min(n, round(n * elapsed / total))
|
||||
groups.append(line_indices[start:max(stop, start)])
|
||||
start = max(stop, start)
|
||||
return groups
|
||||
|
||||
@@ -97,7 +106,7 @@ def _align_section(vocals: Path, sec: tuple[float, float], words: list[tuple[str
|
||||
|
||||
def _run(ctx: Ctx) -> None:
|
||||
ref = normalize_lyrics(ctx.inp.lyrics_text)
|
||||
segments = load_transcript(ctx)
|
||||
segments = sorted(load_transcript(ctx), key=lambda s: s.start)
|
||||
sim = lyrics_similarity(segments, ref)
|
||||
if sim < SIMILARITY_THRESHOLD:
|
||||
raise PipelineError(
|
||||
@@ -105,15 +114,21 @@ def _run(ctx: Ctx) -> None:
|
||||
"wrong lyrics or wrong recording?")
|
||||
# normalised so an inverted span can never collapse a section onto one instant
|
||||
sections = [(min(a, b), max(a, b)) for a, b in vocal_sections(segments)]
|
||||
dropped_lines, missing_spans = classify_lyrics_match(segments, ref)
|
||||
matched = [i for i in range(len(ref.lines)) if i not in set(dropped_lines)]
|
||||
vocals = ctx.work / "vocals.wav"
|
||||
rows = []
|
||||
for sec, line_idxs in zip(sections, _assign_lines_to_sections(ref, sections)):
|
||||
for sec, line_idxs in zip(sections, _assign_lines_to_sections(matched, sections)):
|
||||
words = [(w, li) for li in line_idxs for w in ref.lines[li]]
|
||||
if words:
|
||||
rows.extend(_align_section(vocals, sec, words, ctx.inp.language,
|
||||
ctx.cfg.device))
|
||||
_enforce_monotonic(rows)
|
||||
(ctx.work / "alignment.json").write_text(json.dumps(rows))
|
||||
(ctx.work / "align_meta.json").write_text(json.dumps({
|
||||
"dropped_lines": dropped_lines,
|
||||
"missing_spans": [list(s) for s in missing_spans],
|
||||
}))
|
||||
|
||||
|
||||
def align_stage() -> Stage:
|
||||
@@ -3,6 +3,7 @@ from dataclasses import asdict
|
||||
|
||||
import numpy as np
|
||||
|
||||
from ..lyrics import normalize_lyrics
|
||||
from ..notes import build_notes
|
||||
from ..pipeline import Ctx, Stage
|
||||
|
||||
@@ -10,8 +11,15 @@ from ..pipeline import Ctx, Stage
|
||||
def _run(ctx: Ctx) -> None:
|
||||
rows = json.loads((ctx.work / "alignment.json").read_text())
|
||||
data = np.load(ctx.work / "pitch.npz")
|
||||
meta_path = ctx.work / "align_meta.json"
|
||||
meta = json.loads(meta_path.read_text()) if meta_path.exists() else {}
|
||||
dropped_lines = meta.get("dropped_lines", [])
|
||||
missing_spans = [tuple(s) for s in meta.get("missing_spans", [])]
|
||||
ref_lines = normalize_lyrics(ctx.inp.lyrics_text).lines
|
||||
notes, breaks, warnings = build_notes(
|
||||
rows, data["times"], data["f0"], data["conf"], ctx.inp.language)
|
||||
rows, data["times"], data["f0"], data["conf"], ctx.inp.language,
|
||||
dropped_lines=dropped_lines, missing_spans=missing_spans,
|
||||
ref_lines=ref_lines)
|
||||
(ctx.work / "notes_sec.json").write_text(json.dumps({
|
||||
"notes": [asdict(n) for n in notes],
|
||||
"breaks": breaks,
|
||||
@@ -51,11 +51,15 @@ def _run(ctx: Ctx) -> None:
|
||||
meta = json.loads((ctx.work / "meta.json").read_text())
|
||||
src_video = ctx.work / meta["video_file"]
|
||||
fmt, codec = ffprobe_codec(src_video)
|
||||
# The video is background only: the song's audio is #MP3, and the two are
|
||||
# the same recording. A player that does not mute the video plays both at
|
||||
# once, which is heard as an echo, so the audio track is dropped here.
|
||||
if (fmt, codec) in GOOD_VIDEO:
|
||||
shutil.copyfile(src_video, out / f"{slug}{src_video.suffix}")
|
||||
ffmpeg("-i", str(src_video), "-c", "copy", "-an",
|
||||
str(out / f"{slug}{src_video.suffix}"))
|
||||
else:
|
||||
ffmpeg("-i", str(src_video), "-c:v", "libx264", "-crf", "20",
|
||||
"-preset", "medium", "-c:a", "aac", str(out / f"{slug}.mp4"))
|
||||
"-preset", "medium", "-an", str(out / f"{slug}.mp4"))
|
||||
_set_header(txt, "VIDEO", f"{slug}.mp4")
|
||||
cover_src = ctx.inp.cover_path
|
||||
if cover_src is None and meta.get("thumbnail"):
|
||||
@@ -1,15 +1,30 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from ..models import on_release
|
||||
from ..pipeline import Ctx, Stage
|
||||
|
||||
|
||||
@on_release
|
||||
def unload_model() -> None:
|
||||
"""torchcrepe caches its model as attributes on its own infer function."""
|
||||
torchcrepe = sys.modules.get("torchcrepe")
|
||||
if torchcrepe is None:
|
||||
return
|
||||
for attr in ("model", "capacity"):
|
||||
if hasattr(torchcrepe.infer, attr):
|
||||
delattr(torchcrepe.infer, attr)
|
||||
|
||||
|
||||
def track_pitch(vocals_path: Path, model: str, device: str):
|
||||
import torchaudio
|
||||
import torchcrepe
|
||||
|
||||
waveform, sr = torchaudio.load(str(vocals_path))
|
||||
from ..audio_io import load_wav
|
||||
|
||||
waveform, sr = load_wav(vocals_path)
|
||||
waveform = waveform.mean(0, keepdim=True)
|
||||
waveform = torchaudio.functional.resample(waveform, sr, 16000)
|
||||
f0, periodicity = torchcrepe.predict(
|
||||
@@ -0,0 +1,99 @@
|
||||
import difflib
|
||||
import json
|
||||
|
||||
from ..lyrics import RefLyrics, alignment_token
|
||||
from ..pipeline import Ctx, Stage
|
||||
from ..whisper_client import TranscriptSegment, transcribe
|
||||
|
||||
|
||||
def vocal_sections(segments, min_gap: float = 6.0, pad: float = 1.0):
|
||||
sections = []
|
||||
for s in sorted(segments, key=lambda s: s.start):
|
||||
if sections and s.start - sections[-1][1] < min_gap:
|
||||
sections[-1][1] = max(sections[-1][1], s.end)
|
||||
else:
|
||||
sections.append([s.start, s.end])
|
||||
return [(max(0.0, a - pad), b + pad) for a, b in sections]
|
||||
|
||||
|
||||
def _tokens(text: str) -> list[str]:
|
||||
return [t for w in text.split() if (t := alignment_token(w))]
|
||||
|
||||
|
||||
def match_lyrics(segments, ref: RefLyrics):
|
||||
heard, seg_of = [], []
|
||||
for si, s in enumerate(segments):
|
||||
for t in _tokens(s.text):
|
||||
heard.append(t)
|
||||
seg_of.append(si)
|
||||
pairs = [(t, li) for w, li in zip(ref.words, ref.word_line_index)
|
||||
if (t := alignment_token(w))]
|
||||
expected = [t for t, _ in pairs]
|
||||
line_of = [li for _, li in pairs]
|
||||
matcher = difflib.SequenceMatcher(a=heard, b=expected)
|
||||
return matcher, seg_of, line_of
|
||||
|
||||
|
||||
def lyrics_similarity(segments, ref: RefLyrics) -> float:
|
||||
matcher, _, _ = match_lyrics(segments, ref)
|
||||
return matcher.ratio()
|
||||
|
||||
|
||||
def classify_lyrics_match(segments, ref: RefLyrics, pad: float = 1.0
|
||||
) -> tuple[list[int], list[tuple[float, float]]]:
|
||||
"""Split reference lines into matched vs. dropped, and flag sung spans no
|
||||
reference line explains.
|
||||
|
||||
dropped_lines: reference line indices covered only by 'insert' opcodes
|
||||
(present in the lyrics, never heard). missing_spans: the padded, merged
|
||||
bounds of the heard segments behind 'delete' opcodes (heard, but matching
|
||||
no reference word). A segment can also hold matched lines, so a span
|
||||
marks where to look, not words to discard.
|
||||
"""
|
||||
matcher, seg_of, line_of = match_lyrics(segments, ref)
|
||||
covered = [False] * len(line_of)
|
||||
for tag, _, _, bj1, bj2 in matcher.get_opcodes():
|
||||
if tag in ("equal", "replace"):
|
||||
for j in range(bj1, bj2):
|
||||
covered[j] = True
|
||||
by_line = {}
|
||||
for j, li in enumerate(line_of):
|
||||
by_line.setdefault(li, []).append(covered[j])
|
||||
dropped_lines = sorted(li for li, cov in by_line.items() if not any(cov))
|
||||
|
||||
spans = []
|
||||
for tag, ai1, ai2, _, _ in matcher.get_opcodes():
|
||||
if tag != "delete":
|
||||
continue
|
||||
sis = [seg_of[i] for i in range(ai1, ai2)]
|
||||
start = max(0.0, min(segments[si].start for si in sis) - pad)
|
||||
end = max(segments[si].end for si in sis) + pad
|
||||
spans.append((start, end))
|
||||
return dropped_lines, _merge_spans(spans)
|
||||
|
||||
|
||||
def _merge_spans(spans: list[tuple[float, float]]) -> list[tuple[float, float]]:
|
||||
merged: list[list[float]] = []
|
||||
for a, b in sorted(spans):
|
||||
if merged and a <= merged[-1][1]:
|
||||
merged[-1][1] = max(merged[-1][1], b)
|
||||
else:
|
||||
merged.append([a, b])
|
||||
return [tuple(s) for s in merged]
|
||||
|
||||
|
||||
def _run(ctx: Ctx) -> None:
|
||||
segments = transcribe(ctx.cfg, ctx.work / "vocals.wav", ctx.inp.language)
|
||||
(ctx.work / "transcript.json").write_text(json.dumps(
|
||||
[{"start": s.start, "end": s.end, "text": s.text} for s in segments]))
|
||||
|
||||
|
||||
def transcribe_stage() -> Stage:
|
||||
return Stage(name="transcribe",
|
||||
outputs=lambda ctx: [ctx.work / "transcript.json"],
|
||||
run=_run)
|
||||
|
||||
|
||||
def load_transcript(ctx: Ctx) -> list[TranscriptSegment]:
|
||||
data = json.loads((ctx.work / "transcript.json").read_text())
|
||||
return [TranscriptSegment(**d) for d in data]
|
||||
@@ -4,7 +4,7 @@
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="color-scheme" content="dark">
|
||||
<title>karaoke-maker</title>
|
||||
<title>songtrace</title>
|
||||
<style>
|
||||
@font-face {
|
||||
font-family: "Chakra Petch";
|
||||
@@ -111,6 +111,7 @@ body {
|
||||
}
|
||||
|
||||
#sidebar .eyebrow, #jobs .empty-note { padding: 0 18px; }
|
||||
#create-song { display: block; width: calc(100% - 36px); margin: 0 18px 20px; padding: 9px 12px; }
|
||||
|
||||
#jobs { display: flex; flex-direction: column; gap: 1px; margin-top: 8px; }
|
||||
|
||||
@@ -194,8 +195,13 @@ body {
|
||||
}
|
||||
|
||||
#song-form { display: grid; gap: 18px; }
|
||||
.song-form-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 18px; }
|
||||
.song-form-main { display: grid; gap: 18px; align-content: start; }
|
||||
.song-form-lyrics textarea { height: 100%; flex: 1 1 auto; }
|
||||
.pair { display: grid; grid-template-columns: 1fr 1fr; gap: 18px; }
|
||||
.field { display: flex; flex-direction: column; gap: 6px; }
|
||||
/* [hidden] is only display:none, which the rule above would otherwise win */
|
||||
.field[hidden], .keep-note[hidden] { display: none; }
|
||||
.field > span {
|
||||
font-family: var(--font-display);
|
||||
font-weight: 600;
|
||||
@@ -220,7 +226,6 @@ input:not([type="file"]):hover { border-bottom-color: var(--text-muted); }
|
||||
input:not([type="file"]):focus { border-bottom-color: var(--aqua); }
|
||||
|
||||
textarea {
|
||||
min-height: 8.5rem;
|
||||
padding: 11px 13px;
|
||||
line-height: 1.6;
|
||||
background: var(--panel);
|
||||
@@ -229,6 +234,7 @@ textarea {
|
||||
}
|
||||
textarea:hover { border-color: var(--text-muted); }
|
||||
textarea:focus { border-color: var(--aqua); }
|
||||
.song-form-lyrics textarea { min-height: 8.5rem; }
|
||||
|
||||
input[type="file"] {
|
||||
font-size: 0.82rem;
|
||||
@@ -304,6 +310,7 @@ button { cursor: pointer; }
|
||||
transition: background 120ms ease;
|
||||
}
|
||||
.primary:hover { background: #BE7BF9; }
|
||||
.primary:disabled { background: var(--control); color: var(--text-muted); cursor: not-allowed; }
|
||||
|
||||
#form-error { margin: 12px 0 0; min-height: 1.2em; font-size: 0.85rem; color: var(--fail); }
|
||||
|
||||
@@ -316,20 +323,47 @@ button { cursor: pointer; }
|
||||
}
|
||||
#review h3 span { color: var(--text-muted); font-weight: 400; }
|
||||
|
||||
#warnings { display: flex; flex-direction: column; gap: 1px; }
|
||||
.warn-row {
|
||||
display: grid;
|
||||
grid-template-columns: auto auto 1fr;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 10px 12px;
|
||||
background: var(--panel);
|
||||
/* sticks to the top of #main so the video stays in frame as the lyrics scroll */
|
||||
#player { position: sticky; top: 0; z-index: 2; margin: 0 0 14px; padding: 6px 0 10px; background: var(--bg); }
|
||||
#player-video {
|
||||
display: block;
|
||||
width: 100%;
|
||||
max-height: 30vh;
|
||||
background: #000;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 2px;
|
||||
}
|
||||
#player-video[hidden] { display: none; }
|
||||
/* the native transport renders light otherwise, against everything around it */
|
||||
#player-media { display: block; width: 100%; margin-top: 8px; color-scheme: dark; }
|
||||
|
||||
#lyrics-review { display: flex; flex-direction: column; gap: 10px; }
|
||||
.review-line { padding: 8px 12px; background: var(--panel); border: 1px solid var(--border); }
|
||||
.review-line-active { background: var(--panel-raised); border-color: var(--aqua); }
|
||||
.review-word-sung { color: var(--text-muted); }
|
||||
.review-word-active { color: var(--aqua); font-weight: 600; }
|
||||
.review-playhead { position: absolute; top: 0; bottom: 0; width: 2px; margin-left: -1px; background: var(--aqua); }
|
||||
.review-seek { cursor: pointer; }
|
||||
.review-line-text { display: block; font-size: 0.9rem; }
|
||||
.review-line-dropped { opacity: 0.55; }
|
||||
.review-line-dropped .review-line-text { text-decoration: line-through; }
|
||||
.review-line-note { display: block; margin-top: 4px; font-size: 0.72rem; color: var(--text-muted); }
|
||||
.review-line-timeline { position: relative; margin-top: 4px; }
|
||||
.review-lane { position: absolute; top: 0; bottom: 0; }
|
||||
.review-lane-alt { background: var(--panel-raised); }
|
||||
.review-word-row { position: relative; height: 18px; }
|
||||
.review-word { position: absolute; top: 0; white-space: nowrap; font-size: 0.9rem; }
|
||||
.review-roll { position: relative; display: block; width: 100%; height: 20px; margin-top: 4px; }
|
||||
.review-note { fill: var(--aqua); }
|
||||
.review-note-low { fill: var(--warn); }
|
||||
.review-missing {
|
||||
margin: 0;
|
||||
padding: 6px 12px;
|
||||
font-size: 0.78rem;
|
||||
color: var(--warn);
|
||||
background: var(--panel);
|
||||
border: 1px dashed var(--border);
|
||||
}
|
||||
.warn-row + .warn-row { margin-top: 6px; }
|
||||
.warn-icon { color: var(--warn); }
|
||||
.warn-count { font-size: 0.85rem; }
|
||||
.warn-range { justify-self: end; font-family: var(--font-mono); font-size: 0.82rem; color: var(--text-muted); }
|
||||
|
||||
.review-error { margin: 0 0 14px; font-size: 0.85rem; color: var(--fail); }
|
||||
|
||||
@@ -343,7 +377,7 @@ button { cursor: pointer; }
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
.output-path code { flex: 1; overflow-wrap: anywhere; font-family: var(--font-mono); font-size: 0.82rem; }
|
||||
#copy-path, .action, #cancel-rebuild {
|
||||
#copy-path, .action {
|
||||
flex: none;
|
||||
padding: 5px 12px;
|
||||
font-family: var(--font-display);
|
||||
@@ -356,10 +390,11 @@ button { cursor: pointer; }
|
||||
border: 1px solid var(--control);
|
||||
border-radius: 2px;
|
||||
}
|
||||
#copy-path:hover, .action:hover, #cancel-rebuild:hover { border-color: var(--text-muted); }
|
||||
#copy-path:hover, .action:hover { border-color: var(--text-muted); }
|
||||
|
||||
.form-actions { display: flex; align-items: center; gap: 14px; }
|
||||
.keep-note { margin: -8px 0 0; font-size: 0.78rem; color: var(--aqua); }
|
||||
#form-lock { margin: 0; font-size: 0.78rem; color: var(--text-muted); }
|
||||
#review-actions { display: flex; gap: 10px; margin-top: 14px; }
|
||||
#review-lineage { margin: -10px 0 14px; }
|
||||
.lineage {
|
||||
@@ -464,6 +499,11 @@ button { cursor: pointer; }
|
||||
#stage-track { margin-left: 0; flex-wrap: wrap; }
|
||||
}
|
||||
|
||||
@media (max-width: 720px) {
|
||||
.song-form-grid { grid-template-columns: 1fr; }
|
||||
.song-form-lyrics textarea { height: auto; }
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
*, *::before, *::after { animation: none !important; transition: none !important; }
|
||||
}
|
||||
@@ -473,10 +513,11 @@ button { cursor: pointer; }
|
||||
|
||||
<aside id="sidebar">
|
||||
<div class="brand">
|
||||
<h1 class="wordmark"><span>KARAOKE</span> <span class="second">MAKER</span></h1>
|
||||
<h1 class="wordmark"><span>SONG</span> <span class="second">TRACE</span></h1>
|
||||
<div class="kana" aria-hidden="true">カラオケ</div>
|
||||
</div>
|
||||
<div>
|
||||
<button type="button" id="create-song" class="primary">CREATE SONG</button>
|
||||
<h2 class="eyebrow">QUEUE</h2>
|
||||
<div id="jobs"></div>
|
||||
</div>
|
||||
@@ -486,32 +527,40 @@ button { cursor: pointer; }
|
||||
<section id="new-song">
|
||||
<h2>NEW SONG</h2>
|
||||
<form id="song-form">
|
||||
<div class="pair">
|
||||
<label class="field"><span>Title</span><input name="title" required></label>
|
||||
<label class="field"><span>Artist</span><input name="artist" required></label>
|
||||
</div>
|
||||
<label class="field"><span>Lyrics</span><textarea name="lyrics" required></textarea></label>
|
||||
<details>
|
||||
<summary>Advanced</summary>
|
||||
<div class="stack">
|
||||
<div class="song-form-grid">
|
||||
<div class="song-form-main">
|
||||
<div class="pair">
|
||||
<label class="field"><span>Title</span><input name="title" required></label>
|
||||
<label class="field"><span>Artist</span><input name="artist" required></label>
|
||||
</div>
|
||||
<label class="field"><span>YouTube URL</span><input name="youtube_url"></label>
|
||||
<label class="field"><span>Video file</span><input type="file" name="video"></label>
|
||||
<p class="keep-note" data-keep="video" hidden></p>
|
||||
<div class="pair">
|
||||
<label class="field"><span>Genre</span><input name="genre"></label>
|
||||
<label class="field"><span>Year</span><input name="year" type="number"></label>
|
||||
</div>
|
||||
<div class="pair">
|
||||
<label class="field"><span>Language</span><input name="language"></label>
|
||||
<label class="field"><span>BPM override</span><input name="bpm" type="number" step="any"></label>
|
||||
</div>
|
||||
<label class="field"><span>Cover image</span><input type="file" name="cover"></label>
|
||||
<p class="keep-note" data-keep="cover" hidden></p>
|
||||
<label class="field" id="restart-field" hidden>
|
||||
<span>Restart from</span>
|
||||
<select name="force_from"></select>
|
||||
</label>
|
||||
<details>
|
||||
<summary>Advanced</summary>
|
||||
<div class="stack">
|
||||
<div class="pair">
|
||||
<label class="field"><span>Genre</span><input name="genre"></label>
|
||||
<label class="field"><span>Year</span><input name="year" type="number"></label>
|
||||
</div>
|
||||
<div class="pair">
|
||||
<label class="field"><span>Language</span><input name="language"></label>
|
||||
<label class="field"><span>BPM override</span><input name="bpm" type="number" step="any"></label>
|
||||
</div>
|
||||
<label class="field"><span>Cover image</span><input type="file" name="cover"></label>
|
||||
<p class="keep-note" data-keep="cover" hidden></p>
|
||||
</div>
|
||||
</details>
|
||||
</div>
|
||||
</details>
|
||||
<label class="field song-form-lyrics"><span>Lyrics</span><textarea name="lyrics" required></textarea></label>
|
||||
</div>
|
||||
<div class="form-actions">
|
||||
<button type="submit" class="primary">CREATE SONG</button>
|
||||
<button type="button" id="cancel-rebuild" hidden>CANCEL</button>
|
||||
<p id="form-lock" hidden></p>
|
||||
</div>
|
||||
</form>
|
||||
<p id="form-error" role="alert"></p>
|
||||
@@ -522,7 +571,11 @@ button { cursor: pointer; }
|
||||
<h3 id="review-title"></h3>
|
||||
<p id="review-lineage" hidden></p>
|
||||
<p id="review-error" class="review-error" hidden></p>
|
||||
<div id="warnings"><p class="empty-note">no songs queued yet — nothing to review</p></div>
|
||||
<div id="player" hidden>
|
||||
<video id="player-video" muted playsinline preload="metadata"></video>
|
||||
<audio id="player-media" controls preload="metadata"></audio>
|
||||
</div>
|
||||
<div id="lyrics-review"><p class="empty-note">no songs queued yet — nothing to review</p></div>
|
||||
<div id="output-path"></div>
|
||||
<div id="review-actions"></div>
|
||||
<details>
|
||||
@@ -552,17 +605,20 @@ const nowTitle = document.getElementById("now-title");
|
||||
const stageTrack = document.getElementById("stage-track");
|
||||
const reviewActions = document.getElementById("review-actions");
|
||||
const reviewLineage = document.getElementById("review-lineage");
|
||||
const cancelRebuild = document.getElementById("cancel-rebuild");
|
||||
const createSong = document.getElementById("create-song");
|
||||
const formLock = document.getElementById("form-lock");
|
||||
const formHeading = document.querySelector("#new-song h2");
|
||||
const submitButton = form.querySelector("button[type=submit]");
|
||||
let rebuildOf = null;
|
||||
let formDirty = false;
|
||||
// set by CREATE SONG: the page is a blank sheet on purpose, so the poll must
|
||||
// not quietly select the newest song back into it
|
||||
let composing = false;
|
||||
let JOBS = [];
|
||||
let selectedId = null;
|
||||
let STAGES = [];
|
||||
let pollTimer = null;
|
||||
|
||||
const WARN_RE = /^(\d+)\s+low-confidence notes?\s+near\s+(\d+:\d{2})–(\d+:\d{2})/;
|
||||
|
||||
const STATE = {
|
||||
queued: { glyph: "○", cls: "st-queued" },
|
||||
running: { glyph: "◈", cls: "st-running" },
|
||||
@@ -592,10 +648,16 @@ form.addEventListener("submit", async (e) => {
|
||||
if (!resp.ok) {
|
||||
const body = await resp.json().catch(() => ({}));
|
||||
formError.textContent = body.detail || `error: ${resp.status}`;
|
||||
return; // before the reset below, so a rejected submission is never retyped
|
||||
return;
|
||||
}
|
||||
selectedId = (await resp.json()).id;
|
||||
exitRebuild();
|
||||
composing = false;
|
||||
// the fields stay as submitted: the song just sent is the obvious starting
|
||||
// point for the next one, and nothing has to be retyped to correct it. The
|
||||
// queue's CREATE SONG button is what empties the form.
|
||||
rebuildOf = null;
|
||||
formDirty = false;
|
||||
setFormMode(null, null);
|
||||
refreshJobs();
|
||||
});
|
||||
|
||||
@@ -620,7 +682,25 @@ for (const input of form.querySelectorAll('input[type="file"]')) {
|
||||
});
|
||||
}
|
||||
|
||||
cancelRebuild.addEventListener("click", exitRebuild);
|
||||
// selecting another queue entry reloads the form, so track whether there is
|
||||
// anything of the operator's in there worth warning about first
|
||||
form.addEventListener("input", () => { formDirty = true; });
|
||||
|
||||
// Always available, and the form is usually holding a selected song by now, so
|
||||
// this clears the whole page down to a blank sheet — fields, review, player and
|
||||
// queue selection — and brings the form back into view from wherever playback
|
||||
// scrolled the panel to. Leaving another song's review under an empty form
|
||||
// reads as though the two belong together.
|
||||
createSong.addEventListener("click", () => {
|
||||
if (formDirty && !confirm("Discard your unsaved edits to this form?")) return;
|
||||
composing = true;
|
||||
selectedId = null;
|
||||
for (const card of jobsDiv.querySelectorAll(".job")) card.removeAttribute("aria-current");
|
||||
exitRebuild();
|
||||
clearDetail("nothing selected");
|
||||
document.getElementById("new-song").scrollIntoView({ block: "start", behavior: "smooth" });
|
||||
form.elements.title.focus();
|
||||
});
|
||||
|
||||
function jobCard(job) {
|
||||
const card = document.createElement("button");
|
||||
@@ -695,6 +775,11 @@ function renderJobs(jobs) {
|
||||
async function loadStages() {
|
||||
const resp = await fetch("/api/stages");
|
||||
STAGES = (await resp.json()).stages;
|
||||
// a rebuild redoes only what the edit invalidated, which cannot see a change
|
||||
// in songtrace itself — this is how you redo a stage anyway
|
||||
const select = form.elements.force_from;
|
||||
select.replaceChildren(new Option("only what changed", ""),
|
||||
...STAGES.map(s => new Option(s, s)));
|
||||
}
|
||||
|
||||
function renderStageTrack(job) {
|
||||
@@ -754,19 +839,28 @@ async function refreshJobs() {
|
||||
const resp = await fetch("/api/jobs");
|
||||
const jobs = await resp.json();
|
||||
JOBS = jobs;
|
||||
if (selectedId === null && jobs.length) selectedId = jobs[0].id;
|
||||
// deliberately not via selectJob: the auto-selection on load fills the review
|
||||
// panel but leaves the form blank, so landing on the page is still the way to
|
||||
// start a new song. Clicking a queue entry is what loads its inputs.
|
||||
if (!composing && selectedId === null && jobs.length) selectedId = jobs[0].id;
|
||||
renderJobs(jobs);
|
||||
renderNowBar(jobs);
|
||||
if (selectedId) refreshDetail();
|
||||
else clearDetail(composing ? "nothing selected" : undefined);
|
||||
schedulePoll(jobs);
|
||||
}
|
||||
|
||||
async function selectJob(id) {
|
||||
// the form already holds this job; reloading it would drop edits in progress
|
||||
const reload = id !== rebuildOf;
|
||||
if (reload && formDirty && !confirm("Discard your unsaved edits to this form?")) return;
|
||||
composing = false;
|
||||
selectedId = id;
|
||||
for (const card of jobsDiv.querySelectorAll(".job")) {
|
||||
if (card.dataset.id === id) card.setAttribute("aria-current", "true");
|
||||
else card.removeAttribute("aria-current");
|
||||
}
|
||||
if (reload) await loadInputs(id, JOBS.find(j => j.id === id)?.status);
|
||||
refreshDetail();
|
||||
}
|
||||
|
||||
@@ -777,44 +871,327 @@ function emptyNote(text) {
|
||||
return p;
|
||||
}
|
||||
|
||||
function renderWarnings(job) {
|
||||
const box = document.getElementById("warnings");
|
||||
const SVG_NS = "http://www.w3.org/2000/svg";
|
||||
|
||||
function fmtTime(t) {
|
||||
const s = Math.round(t);
|
||||
return `${Math.floor(s / 60)}:${String(s % 60).padStart(2, "0")}`;
|
||||
}
|
||||
|
||||
function missingSpanNode(span) {
|
||||
const p = document.createElement("p");
|
||||
p.className = "review-missing";
|
||||
p.textContent = `♪ unmatched singing, ${fmtTime(span.start)}–${fmtTime(span.end)}`;
|
||||
return p;
|
||||
}
|
||||
|
||||
function pianoRoll(notes, minPitch, span, t0, t1) {
|
||||
const svg = document.createElementNS(SVG_NS, "svg");
|
||||
svg.setAttribute("class", "review-roll");
|
||||
svg.setAttribute("viewBox", "0 0 100 20");
|
||||
svg.setAttribute("preserveAspectRatio", "none");
|
||||
const dur = Math.max(t1 - t0, 0.001);
|
||||
for (const n of notes) {
|
||||
const rect = document.createElementNS(SVG_NS, "rect");
|
||||
const x = ((n.start - t0) / dur) * 100;
|
||||
const w = Math.max(((n.end - n.start) / dur) * 100, 0.5);
|
||||
const y = 18 - ((n.pitch - minPitch) / span) * 16;
|
||||
rect.setAttribute("x", x.toFixed(2));
|
||||
rect.setAttribute("width", w.toFixed(2));
|
||||
rect.setAttribute("y", y.toFixed(2));
|
||||
rect.setAttribute("height", "2");
|
||||
rect.setAttribute("class", n.low_confidence ? "review-note review-note-low" : "review-note");
|
||||
svg.append(rect);
|
||||
}
|
||||
return svg;
|
||||
}
|
||||
|
||||
// Lays words out along a shared time axis (like a karaoke lyric line) so
|
||||
// each word's notes render directly beneath it instead of as one flat strip.
|
||||
function lineTimeline(words, notes, minPitch, span) {
|
||||
const t0 = words[0].start;
|
||||
const t1 = words[words.length - 1].end;
|
||||
const dur = Math.max(t1 - t0, 0.001);
|
||||
const box = document.createElement("div");
|
||||
box.className = "review-line-timeline";
|
||||
words.forEach((w, i) => {
|
||||
const lane = document.createElement("div");
|
||||
lane.className = "review-lane" + (i % 2 ? " review-lane-alt" : "");
|
||||
lane.style.left = `${((w.start - t0) / dur) * 100}%`;
|
||||
lane.style.width = `${Math.max(((w.end - w.start) / dur) * 100, 0.5)}%`;
|
||||
box.append(lane);
|
||||
});
|
||||
const wordsRow = document.createElement("div");
|
||||
wordsRow.className = "review-word-row";
|
||||
for (const w of words) {
|
||||
const label = document.createElement("span");
|
||||
label.className = "review-word";
|
||||
label.style.left = `${((w.start - t0) / dur) * 100}%`;
|
||||
label.textContent = w.text;
|
||||
wordsRow.append(label);
|
||||
}
|
||||
box.append(wordsRow);
|
||||
if (notes.length) box.append(pianoRoll(notes, minPitch, span, t0, t1));
|
||||
const playhead = document.createElement("div");
|
||||
playhead.className = "review-playhead";
|
||||
playhead.hidden = true;
|
||||
box.append(playhead);
|
||||
return box;
|
||||
}
|
||||
|
||||
function lineNode(line, minPitch, span) {
|
||||
const row = document.createElement("div");
|
||||
row.className = "review-line" + (line.dropped ? " review-line-dropped" : "");
|
||||
if (line.dropped) {
|
||||
const text = document.createElement("span");
|
||||
text.className = "review-line-text";
|
||||
text.textContent = line.text;
|
||||
row.append(text);
|
||||
const note = document.createElement("span");
|
||||
note.className = "review-line-note";
|
||||
note.textContent = "not sung — dropped";
|
||||
row.append(note);
|
||||
return row;
|
||||
}
|
||||
if (line.words.length) {
|
||||
row.append(lineTimeline(line.words, line.notes, minPitch, span));
|
||||
return row;
|
||||
}
|
||||
const text = document.createElement("span");
|
||||
text.className = "review-line-text";
|
||||
text.textContent = line.text;
|
||||
row.append(text);
|
||||
if (line.notes.length) {
|
||||
const t0 = line.notes[0].start;
|
||||
const t1 = Math.max(...line.notes.map(n => n.end));
|
||||
row.append(pianoRoll(line.notes, minPitch, span, t0, t1));
|
||||
}
|
||||
return row;
|
||||
}
|
||||
|
||||
// What playback needs from a rendered line: the time range it was laid out
|
||||
// over, the word labels along it, and the bar that sweeps across it.
|
||||
function lineEntry(row, line) {
|
||||
const laid = line.words.length ? line.words : line.notes;
|
||||
if (!laid.length || line.dropped) return null;
|
||||
const labels = [...row.querySelectorAll(".review-word")];
|
||||
return {
|
||||
row,
|
||||
t0: laid[0].start,
|
||||
t1: Math.max(...laid.map(x => x.end)),
|
||||
words: line.words.map((w, i) => ({ el: labels[i], start: w.start, end: w.end })),
|
||||
playhead: row.querySelector(".review-playhead"),
|
||||
};
|
||||
}
|
||||
|
||||
function buildLyricsReview(data) {
|
||||
const allNotes = data.lines.flatMap(l => l.notes);
|
||||
const pitches = allNotes.map(n => n.pitch);
|
||||
const minPitch = pitches.length ? Math.min(...pitches) : 0;
|
||||
const maxPitch = pitches.length ? Math.max(...pitches) : 0;
|
||||
const span = Math.max(1, maxPitch - minPitch);
|
||||
const spans = [...data.missing_spans].sort((a, b) => a.start - b.start);
|
||||
const nodes = [];
|
||||
const entries = [];
|
||||
let spanIdx = 0;
|
||||
for (const line of data.lines) {
|
||||
const lineStart = line.notes.length ? Math.min(...line.notes.map(n => n.start)) : null;
|
||||
while (spanIdx < spans.length && lineStart !== null && spans[spanIdx].start < lineStart) {
|
||||
nodes.push(missingSpanNode(spans[spanIdx]));
|
||||
spanIdx++;
|
||||
}
|
||||
const row = lineNode(line, minPitch, span);
|
||||
const entry = lineEntry(row, line);
|
||||
if (entry) entries.push(entry);
|
||||
nodes.push(row);
|
||||
}
|
||||
while (spanIdx < spans.length) {
|
||||
nodes.push(missingSpanNode(spans[spanIdx]));
|
||||
spanIdx++;
|
||||
}
|
||||
return { nodes, entries };
|
||||
}
|
||||
|
||||
// The song folder's own media, played against the same seconds the review was
|
||||
// built from, so the highlighted line is the one you are hearing.
|
||||
//
|
||||
// Modelled on the game: #MP3 is the audio and the clock, the video is muted
|
||||
// background slaved to it. Playing the video's own audio alongside the mp3 is
|
||||
// the same recording twice, a few frames apart, which is heard as an echo.
|
||||
const player = document.getElementById("player");
|
||||
const media = document.getElementById("player-media");
|
||||
const video = document.getElementById("player-video");
|
||||
const DRIFT = 0.25; // seconds before the picture is worth re-seeking
|
||||
|
||||
function slaveVideo(force) {
|
||||
if (video.hidden || !video.getAttribute("src")) return;
|
||||
if (force || Math.abs(video.currentTime - media.currentTime) > DRIFT) {
|
||||
video.currentTime = media.currentTime;
|
||||
}
|
||||
}
|
||||
|
||||
for (const event of ["seeked", "ratechange"]) {
|
||||
media.addEventListener(event, () => {
|
||||
video.playbackRate = media.playbackRate;
|
||||
slaveVideo(true);
|
||||
});
|
||||
}
|
||||
media.addEventListener("pause", () => video.pause());
|
||||
let timeline = [];
|
||||
let activeEntry = null;
|
||||
let syncFrame = null;
|
||||
|
||||
// the first line that has not finished yet: the one being sung, or the one
|
||||
// coming up while the track is between lines
|
||||
function entryAt(t) {
|
||||
return timeline.find(e => t <= e.t1) ?? timeline[timeline.length - 1] ?? null;
|
||||
}
|
||||
|
||||
function paintEntry(entry, t) {
|
||||
for (const w of entry.words) {
|
||||
w.el.classList.toggle("review-word-sung", t >= w.end);
|
||||
w.el.classList.toggle("review-word-active", t >= w.start && t < w.end);
|
||||
}
|
||||
if (!entry.playhead) return;
|
||||
const inside = t >= entry.t0 && t <= entry.t1;
|
||||
entry.playhead.hidden = !inside;
|
||||
if (inside) {
|
||||
const frac = (t - entry.t0) / Math.max(entry.t1 - entry.t0, 0.001);
|
||||
entry.playhead.style.left = `${frac * 100}%`;
|
||||
}
|
||||
}
|
||||
|
||||
function clearEntry(entry) {
|
||||
entry.row.classList.remove("review-line-active");
|
||||
if (entry.playhead) entry.playhead.hidden = true;
|
||||
for (const w of entry.words) {
|
||||
w.el.classList.remove("review-word-sung", "review-word-active");
|
||||
}
|
||||
}
|
||||
|
||||
function syncPlayback() {
|
||||
const t = media.currentTime;
|
||||
const entry = entryAt(t);
|
||||
if (entry !== activeEntry) {
|
||||
if (activeEntry) clearEntry(activeEntry);
|
||||
activeEntry = entry;
|
||||
if (entry) {
|
||||
entry.row.classList.add("review-line-active");
|
||||
focusActive();
|
||||
}
|
||||
}
|
||||
if (entry) paintEntry(entry, t);
|
||||
}
|
||||
|
||||
// centred so there is context either side, like a scrolling lyric sheet; only
|
||||
// while running, so it does not fight a reader who has scrolled away to look
|
||||
// at something else
|
||||
function focusActive() {
|
||||
if (!activeEntry || media.paused) return;
|
||||
activeEntry.row.scrollIntoView({ block: "center", behavior: "smooth" });
|
||||
}
|
||||
|
||||
function pumpPlayback() {
|
||||
syncPlayback();
|
||||
slaveVideo(false);
|
||||
syncFrame = media.paused ? null : requestAnimationFrame(pumpPlayback);
|
||||
}
|
||||
|
||||
// timeupdate fires about four times a second, too coarse for a playhead
|
||||
media.addEventListener("play", () => {
|
||||
if (syncFrame === null) syncFrame = requestAnimationFrame(pumpPlayback);
|
||||
if (!video.hidden && video.getAttribute("src")) video.play().catch(() => {});
|
||||
focusActive(); // the line may already be active from a seek while paused
|
||||
});
|
||||
for (const event of ["pause", "seeked", "loadedmetadata"]) {
|
||||
media.addEventListener(event, syncPlayback);
|
||||
}
|
||||
|
||||
function bindTimeline(entries) {
|
||||
if (activeEntry) clearEntry(activeEntry);
|
||||
activeEntry = null;
|
||||
timeline = entries;
|
||||
for (const entry of entries) {
|
||||
entry.row.classList.add("review-seek");
|
||||
entry.row.title = "click to play from here";
|
||||
entry.row.addEventListener("click", (e) => {
|
||||
const word = entry.words.find(w => w.el === e.target);
|
||||
media.currentTime = word ? word.start : entry.t0;
|
||||
if (media.paused) syncPlayback();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Returns true when the folder gave us something to play. #MP3 is required —
|
||||
// it is the audio the game uses, so it is the one the review has to track.
|
||||
function setPlayerSource(job, available) {
|
||||
const src = available?.audio ? `/api/jobs/${job.id}/media/audio` : "";
|
||||
player.hidden = !src;
|
||||
if (media.dataset.src === src) return !!src;
|
||||
media.dataset.src = src;
|
||||
if (src) media.src = src;
|
||||
else { media.removeAttribute("src"); media.load(); }
|
||||
|
||||
video.hidden = !(src && available.video);
|
||||
video.poster = available?.cover ? `/api/jobs/${job.id}/media/cover` : "";
|
||||
if (!video.hidden) video.src = `/api/jobs/${job.id}/media/video`;
|
||||
else { video.removeAttribute("src"); video.load(); }
|
||||
return !!src;
|
||||
}
|
||||
|
||||
function resetPlayer() {
|
||||
// the 3s poll reaches here for every running job; reloading a cleared
|
||||
// element each time would be pure churn
|
||||
if (player.hidden && !timeline.length) return;
|
||||
bindTimeline([]);
|
||||
player.hidden = true;
|
||||
media.pause();
|
||||
video.pause();
|
||||
media.dataset.src = "";
|
||||
for (const el of [media, video]) {
|
||||
el.removeAttribute("src");
|
||||
el.removeAttribute("poster");
|
||||
el.load();
|
||||
}
|
||||
}
|
||||
|
||||
async function renderLyricsReview(job) {
|
||||
const box = document.getElementById("lyrics-review");
|
||||
if (job.status === "failed") {
|
||||
delete box.dataset.jobId;
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("job failed — nothing to review"));
|
||||
return;
|
||||
}
|
||||
if (job.status !== "done") {
|
||||
delete box.dataset.jobId;
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("waiting for this song to finish…"));
|
||||
return;
|
||||
}
|
||||
if (!job.warnings.length) {
|
||||
box.replaceChildren(emptyNote("no low-confidence regions — nothing to review"));
|
||||
if (box.dataset.jobId === job.id) return;
|
||||
const resp = await fetch(`/api/jobs/${job.id}/review`);
|
||||
if (job.id !== selectedId) return;
|
||||
if (!resp.ok) {
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("no review data available"));
|
||||
return;
|
||||
}
|
||||
const rows = job.warnings.map(w => {
|
||||
const m = WARN_RE.exec(w);
|
||||
const row = document.createElement("div");
|
||||
row.className = "warn-row";
|
||||
if (m) {
|
||||
row.setAttribute("data-count", m[1]);
|
||||
row.setAttribute("data-start", m[2]);
|
||||
row.setAttribute("data-end", m[3]);
|
||||
const icon = document.createElement("span");
|
||||
icon.className = "warn-icon";
|
||||
icon.textContent = "⚠";
|
||||
const count = document.createElement("span");
|
||||
count.className = "warn-count";
|
||||
count.textContent = `${m[1]} note${m[1] === "1" ? "" : "s"}`;
|
||||
const range = document.createElement("span");
|
||||
range.className = "warn-range";
|
||||
range.textContent = `${m[2]}–${m[3]}`;
|
||||
row.append(icon, count, range);
|
||||
} else {
|
||||
row.textContent = w;
|
||||
}
|
||||
return row;
|
||||
});
|
||||
box.replaceChildren(...rows);
|
||||
const data = await resp.json();
|
||||
if (job.id !== selectedId) return;
|
||||
box.dataset.jobId = job.id;
|
||||
if (!data.lines.length) {
|
||||
resetPlayer();
|
||||
box.replaceChildren(emptyNote("no lyrics to review"));
|
||||
return;
|
||||
}
|
||||
const { nodes, entries } = buildLyricsReview(data);
|
||||
box.replaceChildren(...nodes);
|
||||
// with nothing to play, a highlighted line and a click-to-seek cursor would
|
||||
// both be promising something the song folder cannot deliver
|
||||
const playable = setPlayerSource(job, data.media);
|
||||
bindTimeline(playable ? entries : []);
|
||||
if (playable) syncPlayback();
|
||||
}
|
||||
|
||||
async function copyPath(path, button) {
|
||||
@@ -874,25 +1251,63 @@ function renderOutputPath(job) {
|
||||
outputPathDiv.replaceChildren(wrap);
|
||||
}
|
||||
|
||||
function setFormMode(inp) {
|
||||
function clearDetail(why = "no songs queued yet") {
|
||||
reviewTitle.replaceChildren();
|
||||
reviewError.hidden = true;
|
||||
reviewError.textContent = "";
|
||||
resetPlayer();
|
||||
// the review box, not an element that no longer exists: getting this wrong
|
||||
// threw partway through and left the rest of the panel showing the old song
|
||||
const box = document.getElementById("lyrics-review");
|
||||
delete box.dataset.jobId;
|
||||
box.replaceChildren(emptyNote(`${why} — nothing to review`));
|
||||
outputPathDiv.replaceChildren();
|
||||
reviewActions.replaceChildren();
|
||||
delete reviewActions.dataset.jobId;
|
||||
reviewLineage.replaceChildren();
|
||||
reviewLineage.hidden = true;
|
||||
delete reviewLineage.dataset.jobId;
|
||||
detail.textContent = `${why} — nothing to log`;
|
||||
}
|
||||
|
||||
// a rebuild is refused with 409 until the job settles, so the form says so
|
||||
// rather than letting the submission fail
|
||||
function setSubmitLock(status) {
|
||||
const locked = rebuildOf !== null && (status === "queued" || status === "running");
|
||||
submitButton.disabled = locked;
|
||||
formLock.textContent = locked ? "editable once this run finishes" : "";
|
||||
formLock.hidden = !locked;
|
||||
}
|
||||
|
||||
function setFormMode(inp, status) {
|
||||
formHeading.textContent = rebuildOf ? "REBUILD" : "NEW SONG";
|
||||
submitButton.textContent = rebuildOf ? "REBUILD SONG" : "CREATE SONG";
|
||||
cancelRebuild.hidden = !rebuildOf;
|
||||
setSubmitLock(status);
|
||||
// only a rebuild has stages to restart from; a new song always runs them all
|
||||
document.getElementById("restart-field").hidden = !rebuildOf;
|
||||
if (!rebuildOf) form.elements.force_from.value = "";
|
||||
for (const [field, key] of [["video", "video_filename"], ["cover", "cover_filename"]]) {
|
||||
const note = form.querySelector(`[data-keep="${field}"]`);
|
||||
const kept = rebuildOf && inp ? inp[key] : null;
|
||||
note.textContent = kept ? `keeping ${kept} — attach a file to replace it` : "";
|
||||
note.hidden = !kept;
|
||||
}
|
||||
// the advanced fields carry into a rebuild whether or not the section is open,
|
||||
// so anything actually set must not sit hidden behind a closed summary
|
||||
form.querySelector("details").open = !!inp?.cover_filename ||
|
||||
["genre", "year", "language", "bpm"].some(n => form.elements[n].value);
|
||||
}
|
||||
|
||||
function exitRebuild() {
|
||||
rebuildOf = null;
|
||||
form.reset();
|
||||
setFormMode(null);
|
||||
formDirty = false;
|
||||
setFormMode(null, null);
|
||||
}
|
||||
|
||||
async function startRebuild(id) {
|
||||
// selecting a queue entry brings its inputs back up, so a song can be adjusted
|
||||
// and rebuilt without being retyped
|
||||
async function loadInputs(id, status) {
|
||||
const resp = await fetch(`/api/jobs/${id}/input`);
|
||||
if (!resp.ok) return;
|
||||
const inp = await resp.json();
|
||||
@@ -903,9 +1318,9 @@ async function startRebuild(id) {
|
||||
if (el && value !== null) el.value = value;
|
||||
}
|
||||
rebuildOf = id;
|
||||
setFormMode(inp);
|
||||
formDirty = false;
|
||||
setFormMode(inp, status);
|
||||
formError.textContent = "";
|
||||
form.elements.title.focus();
|
||||
}
|
||||
|
||||
async function rerunJob(id) {
|
||||
@@ -920,21 +1335,60 @@ async function rerunJob(id) {
|
||||
refreshJobs();
|
||||
}
|
||||
|
||||
async function rebuildFullJob(id) {
|
||||
formError.textContent = "";
|
||||
const resp = await fetch(`/api/jobs/${id}/rebuild-full`, { method: "POST" });
|
||||
if (!resp.ok) {
|
||||
const body = await resp.json().catch(() => ({}));
|
||||
formError.textContent = body.detail || `error: ${resp.status}`;
|
||||
return;
|
||||
}
|
||||
selectedId = (await resp.json()).id;
|
||||
refreshJobs();
|
||||
}
|
||||
|
||||
async function deleteJob(id) {
|
||||
if (!confirm("Delete this song? This removes all its files and cannot be undone.")) return;
|
||||
formError.textContent = "";
|
||||
const resp = await fetch(`/api/jobs/${id}`, { method: "DELETE" });
|
||||
if (!resp.ok) {
|
||||
const body = await resp.json().catch(() => ({}));
|
||||
formError.textContent = body.detail || `error: ${resp.status}`;
|
||||
return;
|
||||
}
|
||||
if (selectedId === id) selectedId = null;
|
||||
// the form is holding a job that no longer exists; a rebuild would 404
|
||||
if (rebuildOf === id) exitRebuild();
|
||||
refreshJobs();
|
||||
}
|
||||
|
||||
const ACTION_LABELS = {
|
||||
rerun: "RERUN",
|
||||
"rebuild-full": "REBUILD FROM SCRATCH",
|
||||
delete: "DELETE",
|
||||
};
|
||||
|
||||
const ACTION_HANDLERS = {
|
||||
rerun: rerunJob,
|
||||
"rebuild-full": rebuildFullJob,
|
||||
delete: deleteJob,
|
||||
};
|
||||
|
||||
function actionButton(action, jobId) {
|
||||
const button = document.createElement("button");
|
||||
button.type = "button";
|
||||
button.className = "action";
|
||||
button.dataset.action = action;
|
||||
button.textContent = action === "rerun" ? "RERUN" : "EDIT & REBUILD";
|
||||
button.addEventListener("click",
|
||||
() => action === "rerun" ? rerunJob(jobId) : startRebuild(jobId));
|
||||
button.textContent = ACTION_LABELS[action];
|
||||
button.addEventListener("click", () => ACTION_HANDLERS[action](jobId));
|
||||
return button;
|
||||
}
|
||||
|
||||
function renderActions(job) {
|
||||
// no controls mid-run: both endpoints answer 409 until the job settles
|
||||
const wanted = job.status === "failed" ? ["rerun", "edit"]
|
||||
: job.status === "done" ? ["edit"] : [];
|
||||
// no controls mid-run: every endpoint answers 409 until the job settles
|
||||
const doneActions = ["rebuild-full", "delete"];
|
||||
const wanted = job.status === "failed" ? ["rerun", ...doneActions]
|
||||
: job.status === "done" ? doneActions : [];
|
||||
const shown = [...reviewActions.children].map(b => b.dataset.action);
|
||||
// rebuilding an unchanged row would detach a focused action on every poll
|
||||
if (reviewActions.dataset.jobId === job.id && shown.join() === wanted.join()) return;
|
||||
@@ -969,7 +1423,8 @@ async function refreshDetail() {
|
||||
reviewTitle.replaceChildren(`${job.title} `, artist);
|
||||
reviewError.hidden = !job.error;
|
||||
reviewError.textContent = job.error ?? "";
|
||||
renderWarnings(job);
|
||||
if (rebuildOf === job.id) setSubmitLock(job.status);
|
||||
renderLyricsReview(job);
|
||||
renderOutputPath(job);
|
||||
renderLineage(job);
|
||||
renderActions(job);
|
||||
@@ -1,3 +1,4 @@
|
||||
import json
|
||||
import math
|
||||
import queue
|
||||
import re
|
||||
@@ -5,7 +6,7 @@ import shutil
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
|
||||
@@ -15,11 +16,15 @@ from fastapi.staticfiles import StaticFiles
|
||||
|
||||
from . import rebuild as rebuild_policy
|
||||
from .config import Config
|
||||
from .pipeline import (STAGE_ORDER, PipelineError, SongInput, run_pipeline,
|
||||
song_slug)
|
||||
from .lyrics import normalize_lyrics
|
||||
from .pipeline import (STAGE_ORDER, PipelineError, SongInput, read_warnings,
|
||||
run_pipeline, song_slug)
|
||||
from .ultrastar import get_header, parse_ultrastar
|
||||
|
||||
STATIC = Path(__file__).parent / "static"
|
||||
STAGE_LINE = re.compile(r"^\[(\w+)\] (?:running|cached, skipping)")
|
||||
# the song folder names its own media, exactly as the game reads it
|
||||
MEDIA_HEADERS = {"audio": "MP3", "video": "VIDEO", "cover": "COVER"}
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -35,6 +40,7 @@ class Job:
|
||||
error: str | None = None
|
||||
inp: SongInput | None = None
|
||||
rebuilt_from: str | None = None
|
||||
force_stage: str | None = None
|
||||
created_at: float = field(default_factory=time.time)
|
||||
started_at: float | None = None
|
||||
finished_at: float | None = None
|
||||
@@ -91,6 +97,35 @@ def _cleanup_uploads(cfg: Config, job_id: str, video_path: Path | None) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _discover_jobs(cfg: Config) -> list[Job]:
|
||||
discovered = []
|
||||
if not cfg.work_dir.exists():
|
||||
return discovered
|
||||
for sidecar in cfg.work_dir.glob("*/input.json"):
|
||||
try:
|
||||
data = json.loads(sidecar.read_text())
|
||||
for key in ("video_path", "cover_path"):
|
||||
if data.get(key) is not None:
|
||||
data[key] = Path(data[key])
|
||||
inp = SongInput(**data)
|
||||
slug = song_slug(inp)
|
||||
out = cfg.output_dir / slug
|
||||
done = out.exists() and any(out.iterdir())
|
||||
mtime = sidecar.stat().st_mtime
|
||||
warnings = read_warnings(sidecar.parent) if done else []
|
||||
except (OSError, ValueError, TypeError):
|
||||
continue
|
||||
discovered.append(Job(
|
||||
id=uuid.uuid4().hex, title=inp.title, artist=inp.artist, inp=inp,
|
||||
status="done" if done else "failed",
|
||||
output_dir=str(out) if done else None,
|
||||
warnings=warnings,
|
||||
error=None if done else "interrupted before completion",
|
||||
created_at=mtime, finished_at=mtime))
|
||||
discovered.sort(key=lambda j: j.created_at)
|
||||
return discovered
|
||||
|
||||
|
||||
def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
app = FastAPI()
|
||||
app.mount("/static", StaticFiles(directory=STATIC), name="static")
|
||||
@@ -99,17 +134,24 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
lock = threading.Lock()
|
||||
todo: queue.Queue[str] = queue.Queue()
|
||||
|
||||
for job in _discover_jobs(cfg):
|
||||
jobs[job.id] = job
|
||||
order.append(job.id)
|
||||
|
||||
def _prepare_rebuild(job: Job, log) -> str | None:
|
||||
"""Relocate cached work and report the stage this rebuild restarts from."""
|
||||
if not job.rebuilt_from:
|
||||
return None
|
||||
return job.force_stage
|
||||
with lock:
|
||||
parent = jobs.get(job.rebuilt_from)
|
||||
if parent is None:
|
||||
return None
|
||||
force_from = rebuild_policy.plan(parent.inp, job.inp)
|
||||
return job.force_stage
|
||||
planned = rebuild_policy.plan(parent.inp, job.inp)
|
||||
changed = rebuild_policy.changed_fields(parent.inp, job.inp)
|
||||
rebuild_policy.relocate_work(parent.inp, job.inp, cfg)
|
||||
# a stage the operator asked for can only widen what plan() decided,
|
||||
# which is how "redo the packaging" reaches work nothing invalidated
|
||||
force_from = rebuild_policy.earliest(planned, job.force_stage)
|
||||
log(f"[rebuild] restarting from {force_from or 'the first incomplete stage'}"
|
||||
f" ({', '.join(changed) if changed else 'nothing'} changed)")
|
||||
return force_from
|
||||
@@ -128,6 +170,15 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
shutil.rmtree(old, ignore_errors=True)
|
||||
log(f"[rebuild] removed superseded {old.name}")
|
||||
|
||||
def _write_sidecar(job: Job) -> None:
|
||||
work = cfg.work_dir / song_slug(job.inp)
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
data = asdict(job.inp)
|
||||
for key in ("video_path", "cover_path"):
|
||||
if data[key] is not None:
|
||||
data[key] = str(data[key])
|
||||
(work / "input.json").write_text(json.dumps(data))
|
||||
|
||||
def worker():
|
||||
while True:
|
||||
job_id = todo.get()
|
||||
@@ -144,6 +195,7 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
|
||||
try:
|
||||
force_from = _prepare_rebuild(job, log)
|
||||
_write_sidecar(job)
|
||||
result = runner(job.inp, cfg, log=log, force_from=force_from)
|
||||
with lock:
|
||||
job.output_dir = str(result.output_dir)
|
||||
@@ -164,10 +216,10 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
|
||||
threading.Thread(target=worker, daemon=True).start()
|
||||
|
||||
def _enqueue(job_id: str, inp: SongInput,
|
||||
rebuilt_from: str | None = None) -> dict:
|
||||
def _enqueue(job_id: str, inp: SongInput, rebuilt_from: str | None = None,
|
||||
force_stage: str | None = None) -> dict:
|
||||
job = Job(id=job_id, title=inp.title, artist=inp.artist, inp=inp,
|
||||
rebuilt_from=rebuilt_from)
|
||||
rebuilt_from=rebuilt_from, force_stage=force_stage)
|
||||
with lock:
|
||||
jobs[job_id] = job
|
||||
order.append(job_id)
|
||||
@@ -196,6 +248,7 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
cover: UploadFile | None = File(None)):
|
||||
youtube_url, genre, language = (youtube_url or None, genre or None,
|
||||
language or None)
|
||||
lyrics = lyrics.replace("\r\n", "\n")
|
||||
has_video = bool(video and video.filename)
|
||||
if bool(youtube_url) == has_video:
|
||||
raise HTTPException(400, "provide exactly one of youtube_url / video")
|
||||
@@ -216,6 +269,12 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
parent = _finished_job(job_id)
|
||||
return _enqueue(uuid.uuid4().hex, parent.inp, rebuilt_from=job_id)
|
||||
|
||||
@app.post("/api/jobs/{job_id}/rebuild-full", status_code=201)
|
||||
def rebuild_full(job_id: str):
|
||||
parent = _finished_job(job_id)
|
||||
return _enqueue(uuid.uuid4().hex, parent.inp, rebuilt_from=job_id,
|
||||
force_stage=STAGE_ORDER[0])
|
||||
|
||||
def _require_source(parent_inp: SongInput, inp: SongInput,
|
||||
force_from: str | None) -> None:
|
||||
"""Refuse a rebuild that will rerun acquire against an upload we no longer have."""
|
||||
@@ -235,13 +294,20 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
lyrics: str = Form(...), youtube_url: str | None = Form(None),
|
||||
genre: str | None = Form(None), year: int | None = Form(None),
|
||||
language: str | None = Form(None), bpm: float | None = Form(None),
|
||||
force_from: str | None = Form(None),
|
||||
video: UploadFile | None = File(None),
|
||||
cover: UploadFile | None = File(None)):
|
||||
parent = _finished_job(job_id)
|
||||
force_from = force_from or None
|
||||
if force_from is not None and force_from not in STAGE_ORDER:
|
||||
raise HTTPException(400, f"unknown stage {force_from!r}")
|
||||
# a cleared field arrives as "" from some clients; "" and None must not
|
||||
# look like different inputs to plan()
|
||||
youtube_url, genre, language = (youtube_url or None, genre or None,
|
||||
language or None)
|
||||
# a browser submits a textarea with CRLF; without this every rebuild
|
||||
# from the page looks like a lyrics edit and redoes the alignment
|
||||
lyrics = lyrics.replace("\r\n", "\n")
|
||||
has_video = bool(video and video.filename)
|
||||
if youtube_url and has_video:
|
||||
raise HTTPException(400, "provide exactly one of youtube_url / video")
|
||||
@@ -259,8 +325,9 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
youtube_url=youtube_url, video_path=video_path,
|
||||
genre=genre, year=year, language=language,
|
||||
cover_path=cover_path, bpm_override=bpm)
|
||||
_require_source(parent.inp, inp, rebuild_policy.plan(parent.inp, inp))
|
||||
return _enqueue(new_id, inp, rebuilt_from=job_id)
|
||||
_require_source(parent.inp, inp, rebuild_policy.earliest(
|
||||
rebuild_policy.plan(parent.inp, inp), force_from))
|
||||
return _enqueue(new_id, inp, rebuilt_from=job_id, force_stage=force_from)
|
||||
|
||||
@app.get("/api/stages")
|
||||
def list_stages():
|
||||
@@ -303,6 +370,87 @@ def create_app(cfg: Config, runner=run_pipeline) -> FastAPI:
|
||||
"video_filename": inp.video_path.name if inp.video_path else None,
|
||||
"cover_filename": inp.cover_path.name if inp.cover_path else None}
|
||||
|
||||
@app.delete("/api/jobs/{job_id}", status_code=204)
|
||||
def delete_job(job_id: str):
|
||||
with lock:
|
||||
job = jobs.get(job_id)
|
||||
if job is None:
|
||||
raise HTTPException(404)
|
||||
if job.status in ("queued", "running"):
|
||||
raise HTTPException(409, "job is still in progress")
|
||||
slug = song_slug(job.inp)
|
||||
if any(other.status in ("queued", "running") and song_slug(other.inp) == slug
|
||||
for other_id, other in jobs.items() if other_id != job_id):
|
||||
raise HTTPException(409, "another job for this song is in progress")
|
||||
jobs.pop(job_id, None)
|
||||
if job_id in order:
|
||||
order.remove(job_id)
|
||||
shutil.rmtree(cfg.work_dir / slug, ignore_errors=True)
|
||||
shutil.rmtree(cfg.output_dir / slug, ignore_errors=True)
|
||||
|
||||
def _resolve_media(job: Job, header: str) -> Path | None:
|
||||
"""The file a song folder's #MP3/#VIDEO/#COVER header points at."""
|
||||
slug = song_slug(job.inp)
|
||||
out = (cfg.output_dir / slug).resolve()
|
||||
txt = out / f"{slug}.txt"
|
||||
if not txt.exists():
|
||||
return None
|
||||
name = get_header(parse_ultrastar(txt.read_text(encoding="utf-8")), header)
|
||||
if not name:
|
||||
return None
|
||||
# a header names a file in the song folder and nothing else
|
||||
path = (out / name).resolve()
|
||||
if path.parent != out or not path.is_file():
|
||||
return None
|
||||
return path
|
||||
|
||||
@app.get("/api/jobs/{job_id}/media/{kind}")
|
||||
def get_media(job_id: str, kind: str):
|
||||
header = MEDIA_HEADERS.get(kind)
|
||||
if header is None:
|
||||
raise HTTPException(404)
|
||||
job = _finished_job(job_id)
|
||||
if job.status != "done":
|
||||
raise HTTPException(404)
|
||||
path = _resolve_media(job, header)
|
||||
if path is None:
|
||||
raise HTTPException(404)
|
||||
return FileResponse(path)
|
||||
|
||||
@app.get("/api/jobs/{job_id}/review")
|
||||
def get_review(job_id: str):
|
||||
job = _finished_job(job_id)
|
||||
if job.status != "done":
|
||||
raise HTTPException(404)
|
||||
work = cfg.work_dir / song_slug(job.inp)
|
||||
notes_file = work / "notes_sec.json"
|
||||
if not notes_file.exists():
|
||||
raise HTTPException(404)
|
||||
notes_data = json.loads(notes_file.read_text())
|
||||
meta_file = work / "align_meta.json"
|
||||
meta = json.loads(meta_file.read_text()) if meta_file.exists() else {}
|
||||
dropped = set(meta.get("dropped_lines", []))
|
||||
missing_spans = [{"start": s, "end": e}
|
||||
for s, e in meta.get("missing_spans", [])]
|
||||
align_file = work / "alignment.json"
|
||||
align_rows = json.loads(align_file.read_text()) if align_file.exists() else []
|
||||
words_by_line: dict[int, list] = {}
|
||||
for r in align_rows:
|
||||
words_by_line.setdefault(r["line_index"], []).append(
|
||||
{"text": r["word"], "start": r["start"], "end": r["end"]})
|
||||
ref = normalize_lyrics(job.inp.lyrics_text)
|
||||
by_line: dict[int, list] = {}
|
||||
for n in notes_data["notes"]:
|
||||
by_line.setdefault(n["line_index"], []).append({
|
||||
"start": n["start"], "end": n["start"] + n["duration"],
|
||||
"pitch": n["pitch"], "low_confidence": n["low_confidence"]})
|
||||
lines = [{"index": i, "text": " ".join(words), "dropped": i in dropped,
|
||||
"notes": by_line.get(i, []), "words": words_by_line.get(i, [])}
|
||||
for i, words in enumerate(ref.lines)]
|
||||
media = {kind: _resolve_media(job, header) is not None
|
||||
for kind, header in MEDIA_HEADERS.items()}
|
||||
return {"lines": lines, "missing_spans": missing_spans, "media": media}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from .config import Config
|
||||
from .languages import LANGUAGES
|
||||
|
||||
|
||||
@dataclass
|
||||
class TranscriptSegment:
|
||||
start: float
|
||||
end: float
|
||||
text: str
|
||||
|
||||
|
||||
def transcribe(cfg: Config, audio_path: Path, language: str | None = None,
|
||||
client: httpx.Client | None = None) -> list[TranscriptSegment]:
|
||||
headers = {}
|
||||
if cfg.whisper_api_key:
|
||||
headers["Authorization"] = f"Bearer {cfg.whisper_api_key}"
|
||||
iso_language = None
|
||||
if language:
|
||||
entry = LANGUAGES.get(language.lower())
|
||||
iso_language = entry.iso if entry else language
|
||||
|
||||
own = client is None
|
||||
client = client or httpx.Client(timeout=600)
|
||||
try:
|
||||
with audio_path.open("rb") as f:
|
||||
if cfg.whisper_api == "asr-webservice":
|
||||
# onerahmet/openai-whisper-asr-webservice: fixed /asr endpoint,
|
||||
# file field is "audio_file", and language/output are query
|
||||
# params rather than form fields - there is no "model" param,
|
||||
# the model is fixed by how that server was started.
|
||||
params = {"output": "json"}
|
||||
if iso_language:
|
||||
params["language"] = iso_language
|
||||
resp = client.post(
|
||||
f"{cfg.whisper_url.rstrip('/')}/asr",
|
||||
headers=headers, params=params,
|
||||
files={"audio_file": (audio_path.name, f, "audio/wav")})
|
||||
else:
|
||||
data = {"model": cfg.whisper_model, "response_format": "verbose_json"}
|
||||
if iso_language:
|
||||
data["language"] = iso_language
|
||||
resp = client.post(
|
||||
f"{cfg.whisper_url.rstrip('/')}/audio/transcriptions",
|
||||
headers=headers, data=data,
|
||||
files={"file": (audio_path.name, f, "audio/wav")})
|
||||
resp.raise_for_status()
|
||||
return [TranscriptSegment(s["start"], s["end"], s["text"].strip())
|
||||
for s in resp.json().get("segments", [])]
|
||||
finally:
|
||||
if own:
|
||||
client.close()
|
||||
@@ -0,0 +1,185 @@
|
||||
"""The one place songtrace touches yt-dlp.
|
||||
|
||||
YouTube breaks old yt-dlp releases faster than songtrace is released, so
|
||||
besides the copy installed with songtrace (the bundled one), newer stable
|
||||
releases are fetched from PyPI into a cache dir at runtime and tried first,
|
||||
with the bundled copy as the fallback. yt-dlp runs in a subprocess and is
|
||||
driven through its command line, its most stable interface, so a cached
|
||||
release never mixes with anything songtrace has imported.
|
||||
"""
|
||||
import importlib.metadata
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
from .config import Config
|
||||
from .pipeline import PipelineError
|
||||
|
||||
# The extras are what make YouTube work; see pyproject.toml.
|
||||
REQUIREMENT = "yt-dlp[default,curl-cffi,deno]"
|
||||
CHECK_INTERVAL = 24 * 3600
|
||||
_lock = threading.Lock()
|
||||
|
||||
|
||||
class YtDlpError(PipelineError):
|
||||
pass
|
||||
|
||||
|
||||
def _key(version: str) -> tuple[int, ...]:
|
||||
return tuple(int(p) for p in version.split("."))
|
||||
|
||||
|
||||
def cache_dir() -> Path:
|
||||
base = os.environ.get("XDG_CACHE_HOME") or Path.home() / ".cache"
|
||||
# Installs hold compiled wheels, so they are only good for one Python.
|
||||
return Path(base) / "songtrace" / f"yt-dlp-py{sys.version_info[0]}{sys.version_info[1]}"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class YtDlp:
|
||||
version: str
|
||||
path: Path | None = None # None for the bundled copy
|
||||
|
||||
def _env(self) -> dict[str, str]:
|
||||
env = dict(os.environ)
|
||||
if self.path:
|
||||
env["PYTHONPATH"] = os.pathsep.join(
|
||||
filter(None, [str(self.path), env.get("PYTHONPATH")]))
|
||||
env["PATH"] = os.pathsep.join(
|
||||
[str(self.path / "bin"), env.get("PATH", "")])
|
||||
return env
|
||||
|
||||
def run(self, args: list[str], log) -> None:
|
||||
"""Runs yt-dlp with `args`, passing each line it prints to `log`."""
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-P", "-m", "yt_dlp", "--ignore-config", *args],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
|
||||
errors="replace", env=self._env())
|
||||
last = error = ""
|
||||
for line in proc.stdout:
|
||||
last = line.rstrip()
|
||||
log(last)
|
||||
if last.startswith("ERROR:"):
|
||||
error = last
|
||||
if proc.wait() == 0:
|
||||
return
|
||||
if proc.returncode == 2:
|
||||
raise YtDlpError(f"yt-dlp {self.version} rejected songtrace's "
|
||||
f"options, so it may be incompatible: {last}")
|
||||
raise YtDlpError(error or f"yt-dlp exited with status {proc.returncode}")
|
||||
|
||||
|
||||
def bundled() -> YtDlp:
|
||||
return YtDlp(importlib.metadata.version("yt-dlp"))
|
||||
|
||||
|
||||
def _latest_release() -> str:
|
||||
r = httpx.get("https://pypi.org/pypi/yt-dlp/json", timeout=15,
|
||||
follow_redirects=True)
|
||||
r.raise_for_status()
|
||||
return r.json()["info"]["version"]
|
||||
|
||||
|
||||
def _pip_install(target: Path, version: str) -> None:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "--quiet", "--no-input",
|
||||
"--disable-pip-version-check", "--target", str(target),
|
||||
f"{REQUIREMENT}=={version}"],
|
||||
capture_output=True, text=True)
|
||||
if proc.returncode:
|
||||
raise YtDlpError(f"pip could not install yt-dlp {version}: "
|
||||
f"{proc.stderr.strip()[-500:]}")
|
||||
|
||||
|
||||
def _install(version: str, log) -> YtDlp:
|
||||
"""Installs `version` into the cache unless it is already there.
|
||||
|
||||
The install is checked to run before it is moved into place, so a
|
||||
directory in the cache is always a usable yt-dlp."""
|
||||
dest = cache_dir() / version
|
||||
if dest.is_dir():
|
||||
return YtDlp(version, dest)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
log(f"installing yt-dlp {version}")
|
||||
tmp = Path(tempfile.mkdtemp(dir=dest.parent, prefix=".tmp-"))
|
||||
try:
|
||||
_pip_install(tmp, version)
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-P", "-m", "yt_dlp", "--version"],
|
||||
capture_output=True, text=True, env=YtDlp(version, tmp)._env())
|
||||
got = proc.stdout.strip()
|
||||
if proc.returncode or _key(got) != _key(version):
|
||||
raise YtDlpError(f"yt-dlp {version} failed its check after install: "
|
||||
f"{(got or proc.stderr.strip())[-500:]}")
|
||||
try:
|
||||
tmp.rename(dest)
|
||||
except OSError:
|
||||
if not dest.is_dir():
|
||||
raise
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
return YtDlp(version, dest)
|
||||
|
||||
|
||||
def _cached() -> list[YtDlp]:
|
||||
root = cache_dir()
|
||||
if not root.is_dir():
|
||||
return []
|
||||
found = [YtDlp(p.name, p) for p in root.iterdir()
|
||||
if p.is_dir() and re.fullmatch(r"\d+(\.\d+)*", p.name)]
|
||||
return sorted(found, key=lambda y: _key(y.version))
|
||||
|
||||
|
||||
def _newest(base: YtDlp, refresh: bool, log) -> YtDlp | None:
|
||||
"""The newest cached release newer than `base`, checking PyPI for a newer
|
||||
one first if that has not been done for a day or `refresh` is set."""
|
||||
marker = cache_dir() / ".checked"
|
||||
if (refresh or not marker.exists()
|
||||
or time.time() - marker.stat().st_mtime > CHECK_INTERVAL):
|
||||
# Recorded even on failure, so a broken release is not reinstalled
|
||||
# for every job; a failed download refreshes regardless.
|
||||
marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
marker.touch()
|
||||
try:
|
||||
latest = _latest_release()
|
||||
if _key(latest) > _key(base.version):
|
||||
_install(latest, log)
|
||||
except Exception as e:
|
||||
log(f"could not update yt-dlp: {e}")
|
||||
newer = [y for y in _cached() if _key(y.version) > _key(base.version)]
|
||||
for old in newer[:-1]:
|
||||
shutil.rmtree(old.path, ignore_errors=True)
|
||||
return newer[-1] if newer else None
|
||||
|
||||
|
||||
def candidates(cfg: Config, log, refresh: bool = False) -> list[YtDlp]:
|
||||
"""The yt-dlp copies to try, in order: the newest release (or the one
|
||||
`cfg.ytdlp_version` pins), then the bundled copy."""
|
||||
base = bundled()
|
||||
want = cfg.ytdlp_version
|
||||
if want == "bundled":
|
||||
return [base]
|
||||
with _lock:
|
||||
if want == "auto":
|
||||
first = _newest(base, refresh, log)
|
||||
elif not re.fullmatch(r"\d+(\.\d+)*", want):
|
||||
raise YtDlpError(f"ytdlp_version must be \"auto\", \"bundled\" or a "
|
||||
f"release such as \"2026.08.19\", not {want!r}")
|
||||
elif _key(want) == _key(base.version):
|
||||
first = None
|
||||
else:
|
||||
try:
|
||||
first = _install(want, log)
|
||||
except Exception as e:
|
||||
log(f"could not install the pinned yt-dlp {want}: {e}")
|
||||
first = None
|
||||
return [first, base] if first else [base]
|
||||
Vendored
+36
@@ -0,0 +1,36 @@
|
||||
#TITLE:Placeholder Chart
|
||||
#ARTIST:Test Artist
|
||||
#LANGUAGE:English
|
||||
#YEAR:2000
|
||||
#CREATOR:songtrace tests
|
||||
#GENRE:Test
|
||||
#COVER:Test Artist - Placeholder Chart [CO].jpg
|
||||
#MP3:Test Artist - Placeholder Chart.mp3
|
||||
#VIDEO:Test Artist - Placeholder Chart.mp4
|
||||
#BPM:217,98
|
||||
#GAP:4880
|
||||
#MEDLEYSTARTBEAT:113
|
||||
#MEDLEYENDBEAT:240
|
||||
: 0 3 16 ah
|
||||
: 5 3 20 ah
|
||||
: 11 3 23 la
|
||||
: 17 3 16 la
|
||||
: 22 28 15 ah
|
||||
: 51 2 16 doo
|
||||
: 54 31 16 ~
|
||||
- 95
|
||||
F 113 3 18 la
|
||||
F 117 3 21 la
|
||||
F 121 3 21 la
|
||||
* 125 3 21 doo
|
||||
* 129 12 19 ~
|
||||
- 150
|
||||
: 155 4 12 Na
|
||||
R 160 4 0 na
|
||||
G 165 4 0 na
|
||||
: 170 6 14 la
|
||||
- 190
|
||||
: 194 3 16 La
|
||||
: 198 3 18 la
|
||||
* 202 10 20 ah
|
||||
E
|
||||
Vendored
+21
@@ -0,0 +1,21 @@
|
||||
#TITLE:Café Placeholder
|
||||
#ARTIST:Tést Artist
|
||||
#LANGUAGE:English
|
||||
#GENRE:Test
|
||||
#YEAR:2000
|
||||
#MP3:Tést Artist - Café Placeholder.mp3
|
||||
#CREATOR:songtrace tests
|
||||
#BPM:256
|
||||
#GAP:49673
|
||||
: 0 4 2 Doo
|
||||
: 6 4 4 doo
|
||||
: 12 8 5 laé
|
||||
- 28 30
|
||||
: 32 4 0 Na
|
||||
: 38 4 -2 na
|
||||
* 44 12 -9 ah
|
||||
- 64
|
||||
F 70 4 3 La
|
||||
: 76 6 -5 la
|
||||
: 84 10 -9 ~
|
||||
E
|
||||
+123
-6
@@ -1,8 +1,8 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import Ctx, SongInput
|
||||
from karaoke_maker.stages import acquire
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import Ctx, SongInput
|
||||
from songtrace.stages import acquire
|
||||
|
||||
def make_ctx(tmp_path, **kw):
|
||||
inp = SongInput(title="T", artist="A", lyrics_text="x", **kw)
|
||||
@@ -24,7 +24,7 @@ def test_local_video(tmp_path, monkeypatch):
|
||||
assert any("audio.wav" in str(a[-1]) for a in calls)
|
||||
|
||||
def test_youtube(tmp_path, monkeypatch):
|
||||
def fake_dl(url, dest):
|
||||
def fake_dl(url, dest, cfg, log):
|
||||
video = dest / "source_video.webm"
|
||||
video.write_bytes(b"v")
|
||||
thumb = dest / "source_video.jpg"
|
||||
@@ -46,7 +46,7 @@ def test_outputs_are_meta_and_audio(tmp_path):
|
||||
|
||||
def test_requires_exactly_one_source(tmp_path):
|
||||
import pytest
|
||||
from karaoke_maker.pipeline import PipelineError
|
||||
from songtrace.pipeline import PipelineError
|
||||
ctx = make_ctx(tmp_path)
|
||||
with pytest.raises(PipelineError):
|
||||
acquire.acquire_stage().run(ctx)
|
||||
@@ -73,7 +73,7 @@ def test_classify_no_thumbnail(tmp_path):
|
||||
|
||||
def test_classify_no_video_raises(tmp_path):
|
||||
import pytest
|
||||
from karaoke_maker.pipeline import PipelineError
|
||||
from songtrace.pipeline import PipelineError
|
||||
(tmp_path / "source_video.jpg").write_bytes(b"t")
|
||||
with pytest.raises(PipelineError):
|
||||
acquire._classify_downloads(tmp_path)
|
||||
@@ -83,3 +83,120 @@ def test_classify_picks_largest_video_when_multiple(tmp_path):
|
||||
(tmp_path / "source_video.mp4").write_bytes(b"much larger contents here")
|
||||
video, _ = acquire._classify_downloads(tmp_path)
|
||||
assert video.name == "source_video.mp4"
|
||||
|
||||
def test_classify_ignores_partials_left_by_a_failed_attempt(tmp_path):
|
||||
(tmp_path / "source_video.webm").write_bytes(b"v")
|
||||
(tmp_path / "source_video.f137.mp4.part").write_bytes(b"a much bigger partial")
|
||||
(tmp_path / "source_video.f251.webm.ytdl").write_bytes(b"resume state")
|
||||
video, _ = acquire._classify_downloads(tmp_path)
|
||||
assert video.name == "source_video.webm"
|
||||
|
||||
def test_ydl_args_default_client_adds_no_player_client(tmp_path):
|
||||
cfg = Config(whisper_url="http://x/v1")
|
||||
args = acquire._ydl_args(cfg, tmp_path, "default", "https://yt/x")
|
||||
assert "--extractor-args" not in args
|
||||
assert "--retries" in args and "--fragment-retries" in args
|
||||
assert args[-2:] == ["--", "https://yt/x"]
|
||||
|
||||
def test_ydl_args_named_client_pot_provider_and_cookies(tmp_path):
|
||||
cfg = Config(whisper_url="http://x/v1",
|
||||
ytdlp_pot_provider_url="http://pot:4416",
|
||||
ytdlp_cookies=tmp_path / "cookies.txt",
|
||||
ytdlp_impersonate="chrome")
|
||||
args = acquire._ydl_args(cfg, tmp_path, "tv", "https://yt/x")
|
||||
joined = " ".join(args)
|
||||
assert "--extractor-args youtube:player_client=tv" in joined
|
||||
assert ("--extractor-args youtubepot-bgutilhttp:base_url=http://pot:4416"
|
||||
in joined)
|
||||
assert f"--cookies {tmp_path / 'cookies.txt'}" in joined
|
||||
assert "--impersonate chrome" in joined
|
||||
|
||||
def test_ydl_args_parse_with_real_yt_dlp(tmp_path):
|
||||
"""Every option songtrace passes must exist in the bundled yt-dlp."""
|
||||
import pytest
|
||||
pytest.importorskip("yt_dlp")
|
||||
from yt_dlp import parse_options
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_pot_provider_url="http://p:1",
|
||||
ytdlp_cookies=tmp_path / "c.txt")
|
||||
parse_options(acquire._ydl_args(cfg, tmp_path, "tv", "https://yt/x"))
|
||||
|
||||
class FakeYtDlp:
|
||||
def __init__(self, version, behaviour, seen):
|
||||
self.version, self._behaviour, self._seen = version, behaviour, seen
|
||||
def run(self, args, log):
|
||||
client = next((a for a in args if a.startswith("youtube:")), None)
|
||||
self._seen.append((self.version, client))
|
||||
self._behaviour(self.version, client)
|
||||
|
||||
def _fake_candidates(monkeypatch, behaviour, versions=("2026.09.01", "2026.08.19"),
|
||||
refreshed=None):
|
||||
"""Stub ytdlp.candidates; `refreshed` is what a refresh turns up."""
|
||||
seen = []
|
||||
def candidates(cfg, log, refresh=False):
|
||||
vs = refreshed if refresh and refreshed else versions
|
||||
return [FakeYtDlp(v, behaviour, seen) for v in vs]
|
||||
monkeypatch.setattr(acquire.ytdlp, "candidates", candidates)
|
||||
return seen
|
||||
|
||||
def test_download_falls_through_to_the_next_client(tmp_path, monkeypatch):
|
||||
from songtrace.ytdlp import YtDlpError
|
||||
def behaviour(version, client):
|
||||
if client is None:
|
||||
raise YtDlpError("ERROR: HTTP Error 403: Forbidden")
|
||||
(tmp_path / "source_video.webm").write_bytes(b"v")
|
||||
seen = _fake_candidates(monkeypatch, behaviour)
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=["default", "tv"])
|
||||
logged = []
|
||||
video, thumb = acquire.download_youtube("https://yt/x", tmp_path, cfg,
|
||||
logged.append)
|
||||
assert video.name == "source_video.webm"
|
||||
assert thumb is None
|
||||
assert seen == [("2026.09.01", None),
|
||||
("2026.09.01", "youtube:player_client=tv")]
|
||||
assert any("403" in m for m in logged)
|
||||
|
||||
def test_download_falls_back_to_the_bundled_copy(tmp_path, monkeypatch):
|
||||
from songtrace.ytdlp import YtDlpError
|
||||
def behaviour(version, client):
|
||||
if version == "2026.09.01":
|
||||
raise YtDlpError("rejected songtrace's options")
|
||||
(tmp_path / "source_video.webm").write_bytes(b"v")
|
||||
seen = _fake_candidates(monkeypatch, behaviour)
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=["default"])
|
||||
acquire.download_youtube("https://yt/x", tmp_path, cfg, lambda m: None)
|
||||
assert [v for v, _ in seen] == ["2026.09.01", "2026.08.19"]
|
||||
|
||||
def test_download_retries_with_a_release_found_after_failing(tmp_path, monkeypatch):
|
||||
from songtrace.ytdlp import YtDlpError
|
||||
def behaviour(version, client):
|
||||
if version != "2026.10.01":
|
||||
raise YtDlpError("ERROR: HTTP Error 403: Forbidden")
|
||||
(tmp_path / "source_video.webm").write_bytes(b"v")
|
||||
seen = _fake_candidates(monkeypatch, behaviour,
|
||||
refreshed=("2026.10.01", "2026.08.19"))
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=["default"])
|
||||
acquire.download_youtube("https://yt/x", tmp_path, cfg, lambda m: None)
|
||||
assert [v for v, _ in seen] == ["2026.09.01", "2026.08.19", "2026.10.01"]
|
||||
|
||||
def test_download_reports_versions_and_clients_when_all_fail(tmp_path, monkeypatch):
|
||||
import pytest
|
||||
from songtrace.pipeline import PipelineError
|
||||
from songtrace.ytdlp import YtDlpError
|
||||
def behaviour(version, client):
|
||||
raise YtDlpError("ERROR: HTTP Error 403: Forbidden")
|
||||
seen = _fake_candidates(monkeypatch, behaviour)
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=["default", "tv"])
|
||||
with pytest.raises(PipelineError) as e:
|
||||
acquire.download_youtube("https://yt/x", tmp_path, cfg, lambda m: None)
|
||||
assert "2026.09.01, 2026.08.19" in str(e.value)
|
||||
assert "default, tv" in str(e.value)
|
||||
assert "403" in str(e.value)
|
||||
assert len(seen) == 4
|
||||
|
||||
def test_download_with_no_configured_clients_still_tries_default(tmp_path, monkeypatch):
|
||||
seen = _fake_candidates(
|
||||
monkeypatch,
|
||||
lambda v, c: (tmp_path / "source_video.mp4").write_bytes(b"v"))
|
||||
cfg = Config(whisper_url="http://x/v1", ytdlp_clients=[])
|
||||
acquire.download_youtube("https://yt/x", tmp_path, cfg, lambda m: None)
|
||||
assert seen == [("2026.09.01", None)]
|
||||
|
||||
+27
-4
@@ -2,10 +2,10 @@ import json
|
||||
|
||||
import pytest
|
||||
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.lyrics import normalize_lyrics
|
||||
from karaoke_maker.pipeline import Ctx, PipelineError, SongInput
|
||||
from karaoke_maker.stages import align
|
||||
from songtrace.config import Config
|
||||
from songtrace.lyrics import normalize_lyrics
|
||||
from songtrace.pipeline import Ctx, PipelineError, SongInput
|
||||
from songtrace.stages import align
|
||||
|
||||
LYRICS = "it's one two three\nfour five six seven eight"
|
||||
|
||||
@@ -137,3 +137,26 @@ def test_words_split_across_sections(tmp_path, monkeypatch):
|
||||
check_invariants(rows, LYRICS)
|
||||
assert rows[3]["end"] <= 5.0
|
||||
assert rows[4]["start"] >= 29.0
|
||||
|
||||
|
||||
def test_align_drops_lines_never_sung(tmp_path, monkeypatch):
|
||||
lyrics = "it's one two three\nfour five six seven eight\nnine ten eleven twelve"
|
||||
transcript = [{"start": 1.0, "end": 4.0,
|
||||
"text": "it's one two three four five six seven eight"}]
|
||||
rows = run_align(tmp_path, monkeypatch, lyrics, transcript)
|
||||
assert {r["line_index"] for r in rows} == {0, 1}
|
||||
meta = json.loads((tmp_path / "w" / "align_meta.json").read_text())
|
||||
assert meta["dropped_lines"] == [2]
|
||||
assert meta["missing_spans"] == []
|
||||
|
||||
|
||||
def test_align_flags_missing_span_for_unmatched_singing(tmp_path, monkeypatch):
|
||||
lyrics = "it's one two three\nfour five six seven eight"
|
||||
transcript = [
|
||||
{"start": 1.0, "end": 4.0, "text": "it's one two three four five six seven eight"},
|
||||
{"start": 20.0, "end": 23.0, "text": "completely unrelated ad lib bridge"},
|
||||
]
|
||||
run_align(tmp_path, monkeypatch, lyrics, transcript)
|
||||
meta = json.loads((tmp_path / "w" / "align_meta.json").read_text())
|
||||
assert meta["dropped_lines"] == []
|
||||
assert meta["missing_spans"] == [[19.0, 24.0]]
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
from songtrace.audio_io import load_wav
|
||||
|
||||
|
||||
def test_load_wav_mono(tmp_path):
|
||||
path = tmp_path / "mono.wav"
|
||||
data = np.linspace(-0.5, 0.5, 1600, dtype=np.float32)
|
||||
sf.write(path, data, 16000)
|
||||
waveform, sr = load_wav(path)
|
||||
assert sr == 16000
|
||||
assert waveform.shape == (1, 1600)
|
||||
assert np.allclose(waveform[0].numpy(), data, atol=1e-4)
|
||||
|
||||
|
||||
def test_load_wav_stereo(tmp_path):
|
||||
path = tmp_path / "stereo.wav"
|
||||
left = np.linspace(-0.5, 0.5, 800, dtype=np.float32)
|
||||
right = np.linspace(0.5, -0.5, 800, dtype=np.float32)
|
||||
sf.write(path, np.stack([left, right], axis=1), 16000)
|
||||
waveform, sr = load_wav(path)
|
||||
assert sr == 16000
|
||||
assert waveform.shape == (2, 800)
|
||||
assert np.allclose(waveform[0].numpy(), left, atol=1e-4)
|
||||
assert np.allclose(waveform[1].numpy(), right, atol=1e-4)
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
from karaoke_maker.cli import main
|
||||
from songtrace.cli import main
|
||||
|
||||
def test_version(capsys):
|
||||
assert main(["--version"]) == 0
|
||||
assert "karaoke-maker" in capsys.readouterr().out
|
||||
assert "songtrace" in capsys.readouterr().out
|
||||
|
||||
@@ -2,9 +2,9 @@ import os
|
||||
|
||||
import pytest
|
||||
|
||||
import karaoke_maker.cli as cli
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import PipelineError, PipelineResult
|
||||
import songtrace.cli as cli
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import PipelineError, PipelineResult
|
||||
|
||||
|
||||
def setup_cfg(monkeypatch, tmp_path):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
from karaoke_maker.config import Config, load_config
|
||||
from songtrace.config import Config, load_config
|
||||
|
||||
|
||||
def test_load_minimal(tmp_path):
|
||||
@@ -24,7 +24,7 @@ def test_paths_and_overrides(tmp_path):
|
||||
def test_env_var(tmp_path, monkeypatch):
|
||||
cfg_file = tmp_path / "c.toml"
|
||||
cfg_file.write_text('whisper_url = "http://env/v1"\n')
|
||||
monkeypatch.setenv("KARAOKE_MAKER_CONFIG", str(cfg_file))
|
||||
monkeypatch.setenv("SONGTRACE_CONFIG", str(cfg_file))
|
||||
assert load_config().whisper_url == "http://env/v1"
|
||||
|
||||
|
||||
|
||||
+9
-9
@@ -4,27 +4,27 @@ import shutil
|
||||
import pytest
|
||||
|
||||
requires_full = pytest.mark.skipif(
|
||||
not os.environ.get("KARAOKE_E2E"),
|
||||
reason="set KARAOKE_E2E=1 (needs GPU, ffmpeg, whisper server, network)")
|
||||
not os.environ.get("SONGTRACE_E2E"),
|
||||
reason="set SONGTRACE_E2E=1 (needs GPU, ffmpeg, whisper server, network)")
|
||||
|
||||
@requires_full
|
||||
def test_full_pipeline(tmp_path):
|
||||
from karaoke_maker.config import load_config
|
||||
from karaoke_maker.pipeline import SongInput, run_pipeline
|
||||
from songtrace.config import load_config
|
||||
from songtrace.pipeline import SongInput, run_pipeline
|
||||
assert shutil.which("ffmpeg"), "ffmpeg required"
|
||||
cfg = load_config()
|
||||
cfg.output_dir = tmp_path / "out"
|
||||
cfg.work_dir = tmp_path / "work"
|
||||
inp = SongInput(
|
||||
title=os.environ["KARAOKE_E2E_TITLE"],
|
||||
artist=os.environ["KARAOKE_E2E_ARTIST"],
|
||||
lyrics_text=open(os.environ["KARAOKE_E2E_LYRICS"]).read(),
|
||||
youtube_url=os.environ.get("KARAOKE_E2E_URL"),
|
||||
title=os.environ["SONGTRACE_E2E_TITLE"],
|
||||
artist=os.environ["SONGTRACE_E2E_ARTIST"],
|
||||
lyrics_text=open(os.environ["SONGTRACE_E2E_LYRICS"]).read(),
|
||||
youtube_url=os.environ.get("SONGTRACE_E2E_URL"),
|
||||
)
|
||||
result = run_pipeline(inp, cfg)
|
||||
txts = list(result.output_dir.glob("*.txt"))
|
||||
assert len(txts) == 1
|
||||
from karaoke_maker.ultrastar import get_header, parse_ultrastar
|
||||
from songtrace.ultrastar import get_header, parse_ultrastar
|
||||
song = parse_ultrastar(txts[0].read_text(encoding="utf-8"))
|
||||
assert get_header(song, "BPM") and get_header(song, "GAP")
|
||||
assert len(song.body) > 20
|
||||
|
||||
+5
-5
@@ -2,11 +2,11 @@ import json
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.grid import scale_bpm, to_ultrastar
|
||||
from karaoke_maker.pipeline import Ctx, PipelineError, SongInput
|
||||
from karaoke_maker.stages import all_stages, grid as grid_stage_mod
|
||||
from karaoke_maker.ultrastar import (
|
||||
from songtrace.config import Config
|
||||
from songtrace.grid import scale_bpm, to_ultrastar
|
||||
from songtrace.pipeline import Ctx, PipelineError, SongInput
|
||||
from songtrace.stages import all_stages, grid as grid_stage_mod
|
||||
from songtrace.ultrastar import (
|
||||
LineBreak, Note, get_header, parse_ultrastar,
|
||||
)
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from karaoke_maker.lyrics import normalize_lyrics, alignment_token
|
||||
from songtrace.lyrics import normalize_lyrics, alignment_token
|
||||
|
||||
RAW = """[Verse 1]
|
||||
It's one two three
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from songtrace import models
|
||||
from songtrace.stages import align, pitch
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def isolated_callbacks(monkeypatch):
|
||||
"""release() with only the callbacks a test registers."""
|
||||
monkeypatch.setattr(models, "_callbacks", [])
|
||||
return models._callbacks
|
||||
|
||||
|
||||
def fake_torch(available=True, initialised=True, emptied=None, reserved=None):
|
||||
"""A torch whose allocator reports `reserved` bytes, then 0 once emptied."""
|
||||
state = {"reserved": 4 * 2 ** 20 if reserved is None else reserved}
|
||||
|
||||
def empty_cache():
|
||||
emptied.append(True)
|
||||
state["reserved"] = 0
|
||||
|
||||
mod = types.ModuleType("torch")
|
||||
mod.cuda = types.SimpleNamespace(
|
||||
is_available=lambda: available,
|
||||
is_initialized=lambda: initialised,
|
||||
memory_reserved=lambda: state["reserved"],
|
||||
empty_cache=empty_cache)
|
||||
return mod
|
||||
|
||||
|
||||
def test_release_without_torch_or_callbacks_does_nothing(monkeypatch,
|
||||
isolated_callbacks):
|
||||
monkeypatch.delitem(sys.modules, "torch", raising=False)
|
||||
assert models.release() == 0
|
||||
|
||||
|
||||
def test_release_runs_every_registered_callback(isolated_callbacks):
|
||||
called = []
|
||||
models.on_release(lambda: called.append("a"))
|
||||
models.on_release(lambda: called.append("b"))
|
||||
models.release()
|
||||
assert called == ["a", "b"]
|
||||
|
||||
|
||||
def test_on_release_returns_the_function_so_it_works_as_a_decorator():
|
||||
def drop():
|
||||
pass
|
||||
assert models.on_release(drop) is drop
|
||||
models._callbacks.remove(drop)
|
||||
|
||||
|
||||
def test_release_empties_the_allocator_when_cuda_is_in_use(monkeypatch,
|
||||
isolated_callbacks):
|
||||
emptied = []
|
||||
monkeypatch.setitem(sys.modules, "torch", fake_torch(emptied=emptied))
|
||||
assert models.release() == 4 * 2 ** 20
|
||||
assert emptied == [True]
|
||||
|
||||
|
||||
def test_release_leaves_the_allocator_alone_without_a_gpu(monkeypatch,
|
||||
isolated_callbacks):
|
||||
emptied = []
|
||||
monkeypatch.setitem(sys.modules, "torch",
|
||||
fake_torch(available=False, emptied=emptied))
|
||||
assert models.release() == 0
|
||||
assert emptied == []
|
||||
|
||||
|
||||
def test_release_does_not_initialise_cuda_just_to_clean_up(monkeypatch,
|
||||
isolated_callbacks):
|
||||
"""A run that never touched the GPU must not create a context on the way out."""
|
||||
emptied = []
|
||||
monkeypatch.setitem(sys.modules, "torch",
|
||||
fake_torch(initialised=False, emptied=emptied))
|
||||
assert models.release() == 0
|
||||
assert emptied == []
|
||||
|
||||
|
||||
def test_align_drops_its_cached_model():
|
||||
align._model_cache["cuda"] = object()
|
||||
align.unload_model()
|
||||
assert align._model_cache == {}
|
||||
|
||||
|
||||
def test_align_registers_its_unload():
|
||||
assert align.unload_model in models._callbacks
|
||||
|
||||
|
||||
def test_pitch_drops_the_model_torchcrepe_cached(monkeypatch):
|
||||
torchcrepe = types.ModuleType("torchcrepe")
|
||||
|
||||
def infer():
|
||||
pass
|
||||
|
||||
infer.model = object()
|
||||
infer.capacity = "full"
|
||||
torchcrepe.infer = infer
|
||||
monkeypatch.setitem(sys.modules, "torchcrepe", torchcrepe)
|
||||
pitch.unload_model()
|
||||
assert not hasattr(infer, "model")
|
||||
assert not hasattr(infer, "capacity")
|
||||
|
||||
|
||||
def test_pitch_unload_is_fine_when_torchcrepe_was_never_imported(monkeypatch):
|
||||
monkeypatch.delitem(sys.modules, "torchcrepe", raising=False)
|
||||
pitch.unload_model()
|
||||
|
||||
|
||||
def test_pitch_unload_is_fine_when_no_model_was_cached(monkeypatch):
|
||||
torchcrepe = types.ModuleType("torchcrepe")
|
||||
torchcrepe.infer = lambda: None
|
||||
monkeypatch.setitem(sys.modules, "torchcrepe", torchcrepe)
|
||||
pitch.unload_model()
|
||||
|
||||
|
||||
def test_pitch_registers_its_unload():
|
||||
assert pitch.unload_model in models._callbacks
|
||||
|
||||
|
||||
def test_release_reports_nothing_freed_when_the_allocator_was_already_empty(
|
||||
monkeypatch, isolated_callbacks):
|
||||
emptied = []
|
||||
monkeypatch.setitem(sys.modules, "torch",
|
||||
fake_torch(emptied=emptied, reserved=0))
|
||||
assert models.release() == 0
|
||||
assert emptied == [True]
|
||||
+48
-9
@@ -2,13 +2,13 @@ import json
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.notes import (
|
||||
from songtrace.config import Config
|
||||
from songtrace.notes import (
|
||||
MIN_NOTE_SEC, TimedNote, build_notes, hz_to_ultrastar, segment_span,
|
||||
syllabify,
|
||||
)
|
||||
from karaoke_maker.pipeline import Ctx, SongInput
|
||||
from karaoke_maker.stages import all_stages, notes as notes_stage_mod
|
||||
from songtrace.pipeline import Ctx, SongInput
|
||||
from songtrace.stages import all_stages, notes as notes_stage_mod
|
||||
|
||||
|
||||
def flat_pitch(duration=10.0, hz=261.63): # C4
|
||||
@@ -78,7 +78,7 @@ def test_syllabify():
|
||||
assert syllabify("reminds", "English") == ["re", "minds"]
|
||||
assert syllabify("a", "English") == ["a"]
|
||||
assert syllabify("xyzzy", None) == ["xyzzy"]
|
||||
assert syllabify("She's", "English") == ["She's"]
|
||||
assert syllabify("It's", "English") == ["It's"]
|
||||
|
||||
|
||||
def test_syllabify_preserves_word_characters():
|
||||
@@ -96,12 +96,12 @@ def test_syllabify_language_without_pyphen_dict_is_not_mishandled_as_english():
|
||||
def test_build_notes_basic_line():
|
||||
times, f0, conf = flat_pitch()
|
||||
rows = [
|
||||
{"word": "She's", "start": 1.0, "end": 1.5, "score": 0.9, "line_index": 0},
|
||||
{"word": "It's", "start": 1.0, "end": 1.5, "score": 0.9, "line_index": 0},
|
||||
{"word": "got", "start": 1.6, "end": 2.0, "score": 0.9, "line_index": 0},
|
||||
{"word": "Now", "start": 5.0, "end": 5.5, "score": 0.9, "line_index": 1},
|
||||
]
|
||||
notes, breaks, warnings = build_notes(rows, times, f0, conf, "English")
|
||||
assert notes[0].text == "She's" # line-initial: no leading space
|
||||
assert notes[0].text == "It's" # line-initial: no leading space
|
||||
assert notes[1].text == " got" # mid-line: leading space
|
||||
assert notes[2].text == "Now" # new line: no leading space
|
||||
assert len(breaks) == 1 and 2.0 <= breaks[0] <= 5.0
|
||||
@@ -278,7 +278,7 @@ def test_notes_stage_writes_json(tmp_path):
|
||||
times, f0, conf = flat_pitch()
|
||||
np.savez(work / "pitch.npz", times=times, f0=f0, conf=conf)
|
||||
(work / "alignment.json").write_text(json.dumps([
|
||||
{"word": "She's", "start": 1.0, "end": 1.5, "score": 0.9, "line_index": 0},
|
||||
{"word": "It's", "start": 1.0, "end": 1.5, "score": 0.9, "line_index": 0},
|
||||
{"word": "gone", "start": 3.0, "end": 3.5, "score": 0.9, "line_index": 1},
|
||||
]))
|
||||
ctx = Ctx(inp=SongInput(title="T", artist="A", lyrics_text="x", language="English"),
|
||||
@@ -287,7 +287,7 @@ def test_notes_stage_writes_json(tmp_path):
|
||||
assert stage.outputs(ctx) == [work / "notes_sec.json"]
|
||||
stage.run(ctx)
|
||||
data = json.loads((work / "notes_sec.json").read_text())
|
||||
assert [n["text"] for n in data["notes"]] == ["She's", "gone"]
|
||||
assert [n["text"] for n in data["notes"]] == ["It's", "gone"]
|
||||
assert data["breaks"] == [2.25]
|
||||
assert data["warnings"] == []
|
||||
assert TimedNote(**data["notes"][0]).pitch == 0
|
||||
@@ -295,3 +295,42 @@ def test_notes_stage_writes_json(tmp_path):
|
||||
|
||||
def test_notes_stage_registered():
|
||||
assert "notes" in [s.name for s in all_stages()]
|
||||
|
||||
|
||||
def test_build_notes_keeps_matched_words_within_missing_span():
|
||||
# A missing span covers whole Whisper segments, so it can hold matched
|
||||
# lines alongside the unmatched singing; those lines must keep their notes.
|
||||
times, f0, conf = flat_pitch()
|
||||
rows = [
|
||||
{"word": "keep", "start": 1.0, "end": 1.4, "score": 0.9, "line_index": 0},
|
||||
{"word": "held", "start": 5.0, "end": 5.4, "score": 0.9, "line_index": 1},
|
||||
]
|
||||
notes, breaks, warnings = build_notes(rows, times, f0, conf, "English",
|
||||
missing_spans=[(4.5, 6.0)])
|
||||
assert [n.text for n in notes] == ["keep", "held"]
|
||||
assert breaks == [(1.4 + 5.0) / 2]
|
||||
assert warnings == ["missing lyrics near 0:04–0:06"]
|
||||
|
||||
|
||||
def test_warnings_report_dropped_lines():
|
||||
times, f0, conf = flat_pitch()
|
||||
rows = [{"word": "keep", "start": 1.0, "end": 1.4, "score": 0.9, "line_index": 0}]
|
||||
ref_lines = [["keep"], ["the", "outro"]]
|
||||
_, _, warnings = build_notes(rows, times, f0, conf, "English",
|
||||
dropped_lines=[1], ref_lines=ref_lines)
|
||||
assert warnings == ['1 line dropped (not sung) near "the outro"']
|
||||
|
||||
|
||||
def test_warnings_report_missing_spans():
|
||||
times, f0, conf = flat_pitch()
|
||||
rows = [{"word": "keep", "start": 1.0, "end": 1.4, "score": 0.9, "line_index": 0}]
|
||||
_, _, warnings = build_notes(rows, times, f0, conf, "English",
|
||||
missing_spans=[(81.0, 89.5)])
|
||||
assert warnings == ["missing lyrics near 1:21–1:29"]
|
||||
|
||||
|
||||
def test_notes_carry_line_index():
|
||||
times, f0, conf = flat_pitch()
|
||||
rows = [{"word": "hey", "start": 1.0, "end": 1.4, "score": 0.9, "line_index": 3}]
|
||||
notes, _, _ = build_notes(rows, times, f0, conf, "English")
|
||||
assert notes[0].line_index == 3
|
||||
|
||||
+25
-7
@@ -4,12 +4,12 @@ from pathlib import Path
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import (
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import (
|
||||
STAGE_ORDER, Ctx, PipelineError, SongInput, run_pipeline, song_slug,
|
||||
)
|
||||
from karaoke_maker.stages import all_stages, package
|
||||
from karaoke_maker.ultrastar import get_header, parse_ultrastar
|
||||
from songtrace.stages import all_stages, package
|
||||
from songtrace.ultrastar import get_header, parse_ultrastar
|
||||
|
||||
SONG_TXT = ("#TITLE:T\n#ARTIST:A\n#MP3:A - T.mp3\n#COVER:A - T.jpg\n"
|
||||
"#VIDEO:A - T.webm\n#BPM:200\n#GAP:0\n: 0 1 0 la\nE\n")
|
||||
@@ -49,19 +49,37 @@ def make_pipeline_inputs(tmp_path, cover=None):
|
||||
return inp, cfg
|
||||
|
||||
|
||||
def test_package_copies_compatible_video(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(package, "ffmpeg", lambda *a: Path(a[-1]).write_bytes(b"m"))
|
||||
def test_package_remuxes_compatible_video_without_reencoding(tmp_path, monkeypatch):
|
||||
calls = []
|
||||
monkeypatch.setattr(package, "ffmpeg",
|
||||
lambda *a: (calls.append(a), Path(a[-1]).write_bytes(b"m")))
|
||||
monkeypatch.setattr(package, "ffprobe_codec", lambda p: ("webm", "vp9"))
|
||||
ctx = make_ctx(tmp_path, cover=make_cover(tmp_path))
|
||||
package.package_stage().run(ctx)
|
||||
out = tmp_path / "out" / "A - T"
|
||||
assert (out / "A - T.txt").exists()
|
||||
assert (out / "A - T.mp3").exists()
|
||||
assert (out / "A - T.webm").read_bytes() == b"v"
|
||||
assert (out / "A - T.webm").exists()
|
||||
video_call = next(c for c in calls if c[-1].endswith("A - T.webm"))
|
||||
assert "-c" in video_call and "copy" in video_call, "the video was re-encoded"
|
||||
img = Image.open(out / "A - T.jpg")
|
||||
assert img.width == img.height
|
||||
|
||||
|
||||
@pytest.mark.parametrize("probe,name", [(("webm", "vp9"), "A - T.webm"),
|
||||
(("avi", "mpeg4"), "A - T.mp4")])
|
||||
def test_package_strips_the_videos_audio_track(tmp_path, monkeypatch, probe, name):
|
||||
"""#MP3 is the song's audio; a video that keeps its own plays as an echo."""
|
||||
calls = []
|
||||
monkeypatch.setattr(package, "ffmpeg",
|
||||
lambda *a: (calls.append(a), Path(a[-1]).write_bytes(b"m")))
|
||||
monkeypatch.setattr(package, "ffprobe_codec", lambda p: probe)
|
||||
ctx = make_ctx(tmp_path, cover=make_cover(tmp_path))
|
||||
package.package_stage().run(ctx)
|
||||
video_call = next(c for c in calls if c[-1].endswith(name))
|
||||
assert "-an" in video_call
|
||||
|
||||
|
||||
def test_package_reencodes_and_rewrites_video_header(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(package, "ffmpeg", lambda *a: Path(a[-1]).write_bytes(b"m"))
|
||||
monkeypatch.setattr(package, "ffprobe_codec", lambda p: ("avi", "mpeg4"))
|
||||
|
||||
+55
-2
@@ -1,7 +1,7 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import SongInput, Stage, run_pipeline, song_slug
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import SongInput, Stage, read_warnings, run_pipeline, song_slug
|
||||
|
||||
def make_cfg(tmp_path):
|
||||
return Config(whisper_url="http://x/v1", output_dir=tmp_path / "out",
|
||||
@@ -46,3 +46,56 @@ def test_warnings_surfaced(tmp_path):
|
||||
st = Stage(name="n", outputs=lambda ctx: [ctx.work / "n.done"], run=run)
|
||||
result = run_pipeline(inp, cfg, log=lambda m: None, stages=[st])
|
||||
assert result.warnings == ["w1"]
|
||||
|
||||
def test_read_warnings_is_empty_without_a_notes_file(tmp_path):
|
||||
assert read_warnings(tmp_path) == []
|
||||
|
||||
def test_read_warnings_reads_the_notes_file(tmp_path):
|
||||
(tmp_path / "notes_sec.json").write_text(json.dumps({"warnings": ["w1", "w2"]}))
|
||||
assert read_warnings(tmp_path) == ["w1", "w2"]
|
||||
|
||||
def test_gpu_is_released_after_every_stage(tmp_path, monkeypatch):
|
||||
"""Not just at the end: a later stage may wait minutes on the Whisper server."""
|
||||
from songtrace import models
|
||||
cfg, inp, calls = make_cfg(tmp_path), make_inp(), []
|
||||
monkeypatch.setattr(models, "release", lambda: calls.append("release"))
|
||||
stages = [fake_stage("s1", calls), fake_stage("s2", calls)]
|
||||
run_pipeline(inp, cfg, log=lambda m: None, stages=stages)
|
||||
assert calls == ["s1", "release", "s2", "release"]
|
||||
|
||||
def test_gpu_is_released_when_a_stage_fails(tmp_path, monkeypatch):
|
||||
import pytest
|
||||
from songtrace import models
|
||||
from songtrace.pipeline import PipelineError
|
||||
cfg, inp, calls = make_cfg(tmp_path), make_inp(), []
|
||||
monkeypatch.setattr(models, "release", lambda: calls.append("release"))
|
||||
def boom(ctx):
|
||||
raise PipelineError("boom")
|
||||
stages = [Stage(name="s1", outputs=lambda ctx: [ctx.work / "s1.done"], run=boom)]
|
||||
with pytest.raises(PipelineError):
|
||||
run_pipeline(inp, cfg, log=lambda m: None, stages=stages)
|
||||
assert calls == ["release"]
|
||||
|
||||
def test_a_cached_stage_does_not_pay_for_a_release(tmp_path, monkeypatch):
|
||||
from songtrace import models
|
||||
cfg, inp, calls = make_cfg(tmp_path), make_inp(), []
|
||||
stages = [fake_stage("s1", calls)]
|
||||
run_pipeline(inp, cfg, log=lambda m: None, stages=stages)
|
||||
calls.clear()
|
||||
monkeypatch.setattr(models, "release", lambda: calls.append("release"))
|
||||
run_pipeline(inp, cfg, log=lambda m: None, stages=stages)
|
||||
assert calls == []
|
||||
|
||||
def test_a_release_that_freed_memory_is_reported_in_the_log(tmp_path, monkeypatch):
|
||||
from songtrace import models
|
||||
cfg, inp, lines = make_cfg(tmp_path), make_inp(), []
|
||||
monkeypatch.setattr(models, "release", lambda: 512 * 2 ** 20)
|
||||
run_pipeline(inp, cfg, log=lines.append, stages=[fake_stage("s1", [])])
|
||||
assert "[s1] released 512 MiB of GPU memory" in lines
|
||||
|
||||
def test_a_release_that_freed_nothing_stays_out_of_the_log(tmp_path, monkeypatch):
|
||||
from songtrace import models
|
||||
cfg, inp, lines = make_cfg(tmp_path), make_inp(), []
|
||||
monkeypatch.setattr(models, "release", lambda: 0)
|
||||
run_pipeline(inp, cfg, log=lines.append, stages=[fake_stage("s1", [])])
|
||||
assert not any("released" in line for line in lines)
|
||||
|
||||
+3
-3
@@ -1,7 +1,7 @@
|
||||
import numpy as np
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import Ctx, SongInput
|
||||
from karaoke_maker.stages import pitch
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import Ctx, SongInput
|
||||
from songtrace.stages import pitch
|
||||
|
||||
|
||||
def test_pitch_stage_writes_npz(tmp_path, monkeypatch):
|
||||
|
||||
+14
-3
@@ -3,9 +3,9 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import STAGE_ORDER, PipelineError, SongInput
|
||||
from karaoke_maker.rebuild import (FIELD_STAGE, changed_fields, plan,
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import STAGE_ORDER, PipelineError, SongInput
|
||||
from songtrace.rebuild import (FIELD_STAGE, changed_fields, earliest, plan,
|
||||
relocate_work, superseded_output)
|
||||
|
||||
|
||||
@@ -124,3 +124,14 @@ def test_superseded_output_refuses_a_path_outside_the_output_root(tmp_path):
|
||||
def test_superseded_output_refuses_the_output_root_itself(tmp_path):
|
||||
root = tmp_path / "out"
|
||||
assert superseded_output(str(root), str(root / "A - T"), root) is None
|
||||
|
||||
|
||||
def test_earliest_picks_the_earlier_stage():
|
||||
assert earliest("package", "separate") == "separate"
|
||||
assert earliest("separate", "package") == "separate"
|
||||
|
||||
|
||||
def test_earliest_ignores_unset_stages():
|
||||
assert earliest(None, "grid") == "grid"
|
||||
assert earliest("grid", None) == "grid"
|
||||
assert earliest(None, None) is None
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import torch
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import Ctx, SongInput
|
||||
from karaoke_maker.stages import separate
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import Ctx, SongInput
|
||||
from songtrace.stages import separate
|
||||
|
||||
|
||||
def test_separate_writes_stems(tmp_path, monkeypatch):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
from karaoke_maker.lyrics import normalize_lyrics
|
||||
from karaoke_maker.whisper_client import TranscriptSegment
|
||||
from karaoke_maker.stages.transcribe import lyrics_similarity, vocal_sections
|
||||
from songtrace.lyrics import normalize_lyrics
|
||||
from songtrace.whisper_client import TranscriptSegment
|
||||
from songtrace.stages.transcribe import classify_lyrics_match, lyrics_similarity, vocal_sections
|
||||
|
||||
def seg(s, e, t=""):
|
||||
return TranscriptSegment(start=s, end=e, text=t)
|
||||
@@ -26,3 +26,41 @@ def test_similarity_low_for_wrong_lyrics():
|
||||
ref = normalize_lyrics("completely different words here entirely")
|
||||
segs = [seg(0, 2, "it's one two three four five six seven eight")]
|
||||
assert lyrics_similarity(segs, ref) < 0.4
|
||||
|
||||
def test_classify_lyrics_match_flags_extra_reference_line():
|
||||
ref = normalize_lyrics("it's one two three\nfour five six seven eight\nnine ten eleven")
|
||||
segs = [seg(1, 4, "it's one two three four five six seven eight")]
|
||||
dropped, missing = classify_lyrics_match(segs, ref)
|
||||
assert dropped == [2]
|
||||
assert missing == []
|
||||
|
||||
def test_classify_lyrics_match_flags_unmatched_singing():
|
||||
ref = normalize_lyrics("it's one two three\nfour five six seven eight")
|
||||
segs = [seg(1, 4, "it's one two three four five six seven eight"),
|
||||
seg(20, 23, "completely unrelated ad lib bridge")]
|
||||
dropped, missing = classify_lyrics_match(segs, ref)
|
||||
assert dropped == []
|
||||
assert missing == [(19.0, 24.0)]
|
||||
|
||||
def test_classify_lyrics_match_keeps_partially_matched_lines():
|
||||
ref = normalize_lyrics("it's one two three\nfour five six seven eight")
|
||||
# line 1 heard with one word garbled ("five" -> "fife"): still a replace/equal
|
||||
# match, so the line stays matched rather than dropped
|
||||
segs = [seg(1, 4, "it's one two three four fife six seven eight")]
|
||||
dropped, _ = classify_lyrics_match(segs, ref)
|
||||
assert dropped == []
|
||||
|
||||
def test_classify_lyrics_match_scopes_missing_span_to_adlib_not_whole_section():
|
||||
# matched line and an unmatched ad-lib close enough together that
|
||||
# vocal_sections would merge them into one section; missing_spans must
|
||||
# stay tight around the ad-lib so the review marks the right place.
|
||||
ref = normalize_lyrics("it's one two three\nfour five six seven eight")
|
||||
segs = [seg(1, 4, "it's one two three four five six seven eight"),
|
||||
seg(6, 8, "completely unrelated ad lib")]
|
||||
sections = vocal_sections(segs)
|
||||
assert sections == [(0.0, 9.0)] # confirms the two segments do merge
|
||||
dropped, missing = classify_lyrics_match(segs, ref)
|
||||
assert dropped == []
|
||||
assert missing == [(5.0, 9.0)]
|
||||
for a, b in missing:
|
||||
assert not (a <= 1.0 < b) # matched line's start (1.0) untouched
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
from pathlib import Path
|
||||
from karaoke_maker.ultrastar import (
|
||||
from songtrace.ultrastar import (
|
||||
Note, LineBreak, parse_ultrastar, serialize_ultrastar,
|
||||
get_header, format_bpm, parse_bpm,
|
||||
)
|
||||
@@ -10,8 +10,8 @@ def load(name):
|
||||
return (FIXTURES / name).read_text(encoding="utf-8", errors="replace")
|
||||
|
||||
def test_parse_headers_and_notes():
|
||||
song = parse_ultrastar(load("sanctuary.txt"))
|
||||
assert get_header(song, "artist") == "Utada Hikaru"
|
||||
song = parse_ultrastar(load("chart_crlf.txt"))
|
||||
assert get_header(song, "artist") == "Test Artist"
|
||||
assert parse_bpm(get_header(song, "BPM")) == 217.98
|
||||
first = song.body[0]
|
||||
assert first == Note(kind=":", beat=0, length=3, pitch=16, text="ah")
|
||||
@@ -19,16 +19,16 @@ def test_parse_headers_and_notes():
|
||||
assert any(n.kind == "F" for n in song.body if isinstance(n, Note))
|
||||
|
||||
def test_note_text_keeps_leading_space():
|
||||
song = parse_ultrastar(load("sanctuary.txt"))
|
||||
song = parse_ultrastar(load("chart_crlf.txt"))
|
||||
notes = [n for n in song.body if isinstance(n, Note)]
|
||||
assert notes[1].text == " ah"
|
||||
|
||||
def test_negative_pitch():
|
||||
song = parse_ultrastar(load("sweet_child.txt"))
|
||||
song = parse_ultrastar(load("chart_latin1.txt"))
|
||||
assert any(n.pitch == -9 for n in song.body if isinstance(n, Note))
|
||||
|
||||
def test_round_trip():
|
||||
for name in ("sweet_child.txt", "sanctuary.txt"):
|
||||
for name in ("chart_latin1.txt", "chart_crlf.txt"):
|
||||
song = parse_ultrastar(load(name))
|
||||
again = parse_ultrastar(serialize_ultrastar(song))
|
||||
assert again == song
|
||||
|
||||
+805
-17
@@ -1,14 +1,16 @@
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.pipeline import STAGE_ORDER, PipelineError, PipelineResult, song_slug
|
||||
from karaoke_maker.web import STAGE_LINE, create_app
|
||||
from songtrace.config import Config
|
||||
from songtrace.pipeline import STAGE_ORDER, PipelineError, PipelineResult, song_slug
|
||||
from songtrace.web import STAGE_LINE, create_app
|
||||
|
||||
|
||||
def make_client(tmp_path, runner):
|
||||
@@ -190,7 +192,7 @@ def test_cover_upload_kept_after_success(tmp_path):
|
||||
|
||||
def test_cleanup_uploads_removes_an_inherited_upload_dir(tmp_path):
|
||||
"""A rebuild's video may live under its parent's id, not its own."""
|
||||
from karaoke_maker.web import _cleanup_uploads
|
||||
from songtrace.web import _cleanup_uploads
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work",
|
||||
output_dir=tmp_path / "out")
|
||||
parent = cfg.work_dir / "uploads" / "parent"
|
||||
@@ -201,7 +203,7 @@ def test_cleanup_uploads_removes_an_inherited_upload_dir(tmp_path):
|
||||
|
||||
|
||||
def test_cleanup_uploads_ignores_paths_outside_the_upload_root(tmp_path):
|
||||
from karaoke_maker.web import _cleanup_uploads
|
||||
from songtrace.web import _cleanup_uploads
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work",
|
||||
output_dir=tmp_path / "out")
|
||||
outside = tmp_path / "somewhere" / "keep.mp4"
|
||||
@@ -212,7 +214,7 @@ def test_cleanup_uploads_ignores_paths_outside_the_upload_root(tmp_path):
|
||||
|
||||
|
||||
def test_cleanup_uploads_tolerates_a_missing_file(tmp_path):
|
||||
from karaoke_maker.web import _cleanup_uploads
|
||||
from songtrace.web import _cleanup_uploads
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work",
|
||||
output_dir=tmp_path / "out")
|
||||
_cleanup_uploads(cfg, "child", cfg.work_dir / "uploads" / "child" / "gone.mp4")
|
||||
@@ -340,13 +342,13 @@ def test_fonts_are_served(tmp_path):
|
||||
|
||||
|
||||
def test_font_licence_is_bundled():
|
||||
from karaoke_maker.web import STATIC
|
||||
from songtrace.web import STATIC
|
||||
licence = (STATIC / "fonts" / "OFL.txt").read_text(encoding="utf-8", errors="replace")
|
||||
assert "SIL OPEN FONT LICENSE" in licence.upper()
|
||||
|
||||
|
||||
def _index_html():
|
||||
from karaoke_maker.web import STATIC
|
||||
from songtrace.web import STATIC
|
||||
return (STATIC / "index.html").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
@@ -359,6 +361,32 @@ def test_index_has_required_hooks():
|
||||
assert f'name="{name}"' in html, name
|
||||
|
||||
|
||||
def test_video_source_fields_are_outside_advanced_details():
|
||||
html = _index_html()
|
||||
form_start = html.index('id="song-form"')
|
||||
details_start = html.index("<details>", form_start)
|
||||
before = html[form_start:details_start]
|
||||
assert 'name="youtube_url"' in before
|
||||
assert 'name="video"' in before
|
||||
after = html[details_start:]
|
||||
for name in ("genre", "year", "language", "bpm", "cover"):
|
||||
assert f'name="{name}"' in after
|
||||
|
||||
|
||||
def test_song_form_lyrics_uses_a_two_column_grid():
|
||||
css = _page_css(_index_html())
|
||||
assert re.search(r"\.song-form-grid\s*\{[^}]*grid-template-columns:\s*1fr 1fr",
|
||||
css, re.S)
|
||||
|
||||
|
||||
def test_song_form_grid_stacks_on_narrow_screens():
|
||||
css = _page_css(_index_html())
|
||||
assert re.search(
|
||||
r"@media \(max-width: 720px\)\s*\{[^}]*\.song-form-grid\s*\{[^}]*"
|
||||
r"grid-template-columns:\s*1fr\s*;",
|
||||
css, re.S)
|
||||
|
||||
|
||||
def test_index_makes_no_external_requests():
|
||||
import re as _re
|
||||
html = _index_html()
|
||||
@@ -383,10 +411,53 @@ def test_index_respects_reduced_motion():
|
||||
assert "prefers-reduced-motion" in _index_html()
|
||||
|
||||
|
||||
def test_index_parses_warnings_with_endash():
|
||||
def test_index_has_lyrics_review_hook():
|
||||
html = _index_html()
|
||||
assert "–" in html, "warning parser must handle the en-dash separator"
|
||||
assert "data-start" in html and "data-end" in html
|
||||
assert 'id="lyrics-review"' in html
|
||||
assert 'id="warnings"' not in html
|
||||
|
||||
|
||||
def test_index_review_uses_svg_not_canvas():
|
||||
html = _index_html()
|
||||
assert "createElementNS" in html
|
||||
assert "<canvas" not in html
|
||||
|
||||
|
||||
def test_index_review_fetches_review_endpoint():
|
||||
html = _index_html()
|
||||
assert "/review" in html
|
||||
|
||||
|
||||
def test_index_lyrics_review_uses_endash_for_time_ranges():
|
||||
html = _index_html()
|
||||
body = _fn_body(html, "missingSpanNode")
|
||||
assert "–" in body
|
||||
|
||||
|
||||
def test_index_review_lays_out_words_in_time():
|
||||
"""Words must be positioned by their sung timing (percentage-based), not
|
||||
packed as plain flow text, so notes can line up under their own word."""
|
||||
html = _index_html()
|
||||
body = _fn_body(html, "lineTimeline")
|
||||
assert "review-word" in body
|
||||
assert "style.left" in body
|
||||
|
||||
|
||||
def test_lyrics_review_cache_clears_on_job_status_change():
|
||||
"""Regression: switching between done/failed/running jobs must clear the
|
||||
cache so a previously-viewed done job re-fetches when re-selected."""
|
||||
html = _index_html()
|
||||
body = _fn_body(html, "renderLyricsReview")
|
||||
# The "failed" branch must clear the cache before rendering the placeholder
|
||||
assert re.search(
|
||||
r'if\s*\(\s*job\.status\s*===\s*"failed"\s*\)\s*\{[^}]*'
|
||||
r'delete\s+box\.dataset\.jobId',
|
||||
body, re.S), "failed branch must delete cache key"
|
||||
# The "not done" (queued/running) branch must clear the cache before rendering
|
||||
assert re.search(
|
||||
r'if\s*\(\s*job\.status\s*!==\s*"done"\s*\)\s*\{[^}]*'
|
||||
r'delete\s+box\.dataset\.jobId',
|
||||
body, re.S), "not done branch must delete cache key"
|
||||
|
||||
|
||||
def test_index_has_copy_path_control():
|
||||
@@ -513,11 +584,19 @@ def _submit_body(html):
|
||||
return html[start:html.index("\n});\n", start)]
|
||||
|
||||
|
||||
def _listener_body(html, prefix):
|
||||
start = html.index(prefix)
|
||||
return html[start:html.index("\n});\n", start)]
|
||||
|
||||
|
||||
def _rebuild_flaws(html):
|
||||
"""Ways rebuild mode could quietly stop working."""
|
||||
flaws, submit = [], _submit_body(html)
|
||||
if "rebuildOf ?" not in submit or "/rebuild" not in submit:
|
||||
flaws.append("the form always posts to /api/jobs, so edits create a new song")
|
||||
if "form.reset()" in submit or "exitRebuild()" in submit:
|
||||
flaws.append("a successful submission empties the form, so correcting "
|
||||
"or varying the song just sent means retyping it")
|
||||
|
||||
actions = _fn_body(html, "renderActions")
|
||||
if not re.search(r"if \(.*dataset\.jobId === job\.id.*\) return;", actions):
|
||||
@@ -525,12 +604,128 @@ def _rebuild_flaws(html):
|
||||
if '"failed"' not in actions or '"done"' not in actions:
|
||||
flaws.append("the action row does not key off the job status")
|
||||
|
||||
start = _fn_body(html, "startRebuild")
|
||||
if "/input" not in start or "form.elements[" not in start:
|
||||
load = _fn_body(html, "loadInputs")
|
||||
if "/input" not in load or "form.elements[" not in load:
|
||||
flaws.append("rebuild mode does not prefill the form from the job's input")
|
||||
|
||||
select = _fn_body(html, "selectJob")
|
||||
if "loadInputs(" not in select:
|
||||
flaws.append("selecting a queue entry does not bring its inputs back up")
|
||||
if "formDirty" not in select or "confirm(" not in select:
|
||||
flaws.append("selecting a queue entry discards unsaved edits without asking")
|
||||
|
||||
if "if (rebuildOf === id) exitRebuild();" not in _fn_body(html, "deleteJob"):
|
||||
flaws.append("deleting the loaded job leaves the form pointed at it")
|
||||
|
||||
# selecting a queue entry loads it, so the sidebar button is now the only
|
||||
# way back to a blank form
|
||||
create = _listener_body(html, 'createSong.addEventListener("click"')
|
||||
if "exitRebuild()" not in create:
|
||||
flaws.append("the queue's CREATE SONG button does not clear the form")
|
||||
if "formDirty" not in create or "confirm(" not in create:
|
||||
flaws.append("the queue's CREATE SONG button discards edits without asking")
|
||||
if "clearDetail(" not in create:
|
||||
flaws.append("CREATE SONG empties the form but leaves the previous "
|
||||
"song's review and player sitting under it")
|
||||
if "composing" not in create or "composing" not in _fn_body(html, "refreshJobs"):
|
||||
flaws.append("the blank page CREATE SONG made is selected back over by "
|
||||
"the next poll of the queue")
|
||||
|
||||
mode = _fn_body(html, "setFormMode")
|
||||
if 'querySelector("details").open' not in mode:
|
||||
flaws.append("prefilled genre/year/language/bpm stay hidden behind the "
|
||||
"closed Advanced summary, so they rebuild unseen")
|
||||
if "submitButton.disabled" not in _fn_body(html, "setSubmitLock"):
|
||||
flaws.append("the form lets you submit a rebuild the API will answer 409")
|
||||
return flaws
|
||||
|
||||
|
||||
def _rebuild_full_and_delete_flaws(html):
|
||||
flaws = []
|
||||
actions = _fn_body(html, "renderActions")
|
||||
for action in ('"rebuild-full"', '"delete"'):
|
||||
if action not in actions:
|
||||
flaws.append(f"{action} is not offered from renderActions")
|
||||
|
||||
rf = _fn_body(html, "rebuildFullJob")
|
||||
if "/rebuild-full" not in rf or 'method: "POST"' not in rf:
|
||||
flaws.append("rebuildFullJob does not POST to /rebuild-full")
|
||||
|
||||
d = _fn_body(html, "deleteJob")
|
||||
confirm_idx = d.find("confirm(")
|
||||
fetch_idx = d.find("fetch(")
|
||||
if confirm_idx == -1:
|
||||
flaws.append("deleteJob does not confirm before deleting")
|
||||
elif fetch_idx != -1 and confirm_idx > fetch_idx:
|
||||
flaws.append("deleteJob fetches before confirming")
|
||||
if 'method: "DELETE"' not in d:
|
||||
flaws.append("deleteJob does not send a DELETE request")
|
||||
return flaws
|
||||
|
||||
|
||||
def test_rebuild_full_and_delete_controls_are_wired_up():
|
||||
assert _rebuild_full_and_delete_flaws(_index_html()) == []
|
||||
|
||||
|
||||
def _empty_state_flaws(html):
|
||||
refresh = _fn_body(html, "refreshJobs")
|
||||
if "clearDetail(" not in refresh:
|
||||
return ["refreshJobs never resets the panel when no job is selected"]
|
||||
return []
|
||||
|
||||
|
||||
def test_review_panel_resets_when_no_job_is_selected():
|
||||
assert _empty_state_flaws(_index_html()) == []
|
||||
|
||||
|
||||
def _playback_flaws(html):
|
||||
"""Ways the review could stop tracking the media element."""
|
||||
flaws = []
|
||||
source = _fn_body(html, "setPlayerSource")
|
||||
if "media/audio" not in source:
|
||||
flaws.append("the player is not pointed at the job's packaged audio, "
|
||||
"which is the file the game plays and the notes track")
|
||||
if "media/video" not in source:
|
||||
flaws.append("the packaged video is never shown alongside the audio")
|
||||
|
||||
# the mp3 and the video carry the same recording; playing both is an echo
|
||||
if not re.search(r"<video[^>]*\bmuted\b", html):
|
||||
flaws.append("the background video is not muted, so its own audio "
|
||||
"plays on top of the mp3")
|
||||
pump = _fn_body(html, "pumpPlayback")
|
||||
if "slaveVideo(" not in pump:
|
||||
flaws.append("the picture is never pulled back onto the audio clock, "
|
||||
"so it drifts out of sync over a long song")
|
||||
|
||||
if "requestAnimationFrame" not in _fn_body(html, "pumpPlayback"):
|
||||
flaws.append("sync is not driven by the frame loop, so the playhead "
|
||||
"steps at timeupdate's ~4Hz instead of following the audio")
|
||||
|
||||
if "media.paused" not in _fn_body(html, "focusActive"):
|
||||
flaws.append("the view is scrolled even while paused, fighting a reader "
|
||||
"who scrolled away to look at something else")
|
||||
|
||||
sync = _fn_body(html, "syncPlayback")
|
||||
if "clearEntry(" not in sync:
|
||||
flaws.append("the line the playhead left keeps its highlight")
|
||||
clear = _fn_body(html, "clearEntry")
|
||||
for name in ("review-word-sung", "review-word-active", "review-line-active"):
|
||||
if name not in clear:
|
||||
flaws.append(f"{name} is never taken off a line the playhead left")
|
||||
|
||||
# every branch that gives up on rendering a review has to unload the media
|
||||
# too, or the previous song stays loaded and playable under the new one
|
||||
render = _fn_body(html, "renderLyricsReview")
|
||||
if render.count("resetPlayer()") < render.count("emptyNote("):
|
||||
flaws.append("a review the page cannot render leaves the previous "
|
||||
"song's media loaded in the player")
|
||||
return flaws
|
||||
|
||||
|
||||
def test_review_playback_tracks_the_media_element():
|
||||
assert _playback_flaws(_index_html()) == []
|
||||
|
||||
|
||||
# each mutation is a plausible regression; the checker above must reject all of them
|
||||
MUTATIONS = [
|
||||
(_focus_flaws, "queue: card moved unconditionally",
|
||||
@@ -559,6 +754,57 @@ MUTATIONS = [
|
||||
(_rebuild_flaws, "rebuild: action row rebuilt on every poll",
|
||||
"if (reviewActions.dataset.jobId === job.id && shown.join() === wanted.join()) return;",
|
||||
""),
|
||||
(_rebuild_full_and_delete_flaws, "delete: confirmation dialog removed",
|
||||
'if (!confirm("Delete this song? This removes all its files and cannot be undone.")) return;',
|
||||
""),
|
||||
(_rebuild_full_and_delete_flaws, "rebuild-full: no longer offered on a done job",
|
||||
'["rebuild-full", "delete"]', '["delete"]'),
|
||||
(_rebuild_flaws, "prefill: selecting a queue entry no longer loads its inputs",
|
||||
" if (reload) await loadInputs(id, JOBS.find(j => j.id === id)?.status);\n", ""),
|
||||
(_rebuild_flaws, "prefill: unsaved edits discarded without asking",
|
||||
' if (reload && formDirty && !confirm("Discard your unsaved edits to this form?")) return;\n',
|
||||
""),
|
||||
(_rebuild_flaws, "prefill: deleting the loaded job leaves the form pointed at it",
|
||||
" if (rebuildOf === id) exitRebuild();\n", ""),
|
||||
(_rebuild_flaws, "prefill: advanced fields stay hidden behind a closed summary",
|
||||
'form.querySelector("details").open =', "const unusedAdvancedOpen ="),
|
||||
(_rebuild_flaws, "prefill: an in-progress job can be submitted for rebuild anyway",
|
||||
"submitButton.disabled = locked;", ""),
|
||||
(_rebuild_flaws, "submit: a successful submission empties the form again",
|
||||
" rebuildOf = null;\n formDirty = false;\n setFormMode(null, null);\n refreshJobs();",
|
||||
" exitRebuild();\n refreshJobs();"),
|
||||
(_rebuild_flaws, "create: the queue button no longer clears the form",
|
||||
" exitRebuild();\n clearDetail(\"nothing selected\");\n", ""),
|
||||
(_rebuild_flaws, "create: the review is left behind under a blank form",
|
||||
' clearDetail("nothing selected");\n', ""),
|
||||
(_rebuild_flaws, "create: the next poll selects a song back into the blank page",
|
||||
" composing = true;\n", ""),
|
||||
(_rebuild_flaws, "create: the queue button discards edits without asking",
|
||||
' if (formDirty && !confirm("Discard your unsaved edits to this form?")) return;\n composing = true;',
|
||||
" composing = true;"),
|
||||
(_playback_flaws, "playback: sync falls back to the timeupdate event's ~4Hz",
|
||||
"syncFrame = media.paused ? null : requestAnimationFrame(pumpPlayback);",
|
||||
"syncFrame = null;"),
|
||||
(_playback_flaws, "playback: the view is scrolled even while paused",
|
||||
"if (!activeEntry || media.paused) return;", "if (!activeEntry) return;"),
|
||||
(_playback_flaws, "playback: the line the playhead left keeps its highlight",
|
||||
" if (activeEntry) clearEntry(activeEntry);\n", ""),
|
||||
(_playback_flaws, "playback: a word stays lit after the playhead moves on",
|
||||
'w.el.classList.remove("review-word-sung", "review-word-active");',
|
||||
'w.el.classList.remove("review-word-sung");'),
|
||||
(_playback_flaws, "playback: an unrenderable review leaves the old song loaded",
|
||||
" resetPlayer();\n", ""),
|
||||
(_playback_flaws, "playback: the player is not pointed at the packaged audio",
|
||||
'const src = available?.audio ? `/api/jobs/${job.id}/media/audio` : "";',
|
||||
'const src = "";'),
|
||||
(_playback_flaws, "playback: the background video plays its own audio too",
|
||||
'<video id="player-video" muted playsinline',
|
||||
'<video id="player-video" playsinline'),
|
||||
(_playback_flaws, "playback: the picture is never re-synced to the audio",
|
||||
" slaveVideo(false);\n", ""),
|
||||
(_empty_state_flaws, "empty state: clearDetail no longer wired into refreshJobs",
|
||||
"if (selectedId) refreshDetail();\n else clearDetail(composing ? \"nothing selected\" : undefined);",
|
||||
"if (selectedId) refreshDetail();"),
|
||||
]
|
||||
|
||||
|
||||
@@ -594,7 +840,7 @@ def test_rebuild_mode_reuses_the_one_form():
|
||||
|
||||
def test_index_has_rebuild_hooks():
|
||||
html = _index_html()
|
||||
for hook in ("review-actions", "review-lineage", "cancel-rebuild",
|
||||
for hook in ("review-actions", "review-lineage", "create-song",
|
||||
'data-keep="video"', 'data-keep="cover"'):
|
||||
assert hook in html, hook
|
||||
|
||||
@@ -776,7 +1022,7 @@ def test_rebuild_renames_the_work_dir_on_a_retitle(tmp_path):
|
||||
seen = []
|
||||
client, cfg = make_client(tmp_path, capture_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
(cfg.work_dir / "A - T" / "audio.wav").write_bytes(b"x")
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild",
|
||||
data=rebuild_form(title="T2")).json()["id"]
|
||||
@@ -789,7 +1035,7 @@ def test_rebuild_renames_the_work_dir_on_a_retitle(tmp_path):
|
||||
def test_rebuild_fails_the_job_on_a_work_dir_collision(tmp_path):
|
||||
client, cfg = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
(cfg.work_dir / "A - T2").mkdir(parents=True)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild",
|
||||
data=rebuild_form(title="T2")).json()["id"]
|
||||
@@ -817,7 +1063,7 @@ def test_rebuild_inherits_the_parents_source(tmp_path):
|
||||
files={"video": ("v.mp4", io.BytesIO(b"data"), "video/mp4")})
|
||||
parent = resp.json()["id"]
|
||||
wait_status(client, parent, "done")
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
for name in ("meta.json", "audio.wav"):
|
||||
(cfg.work_dir / "A - T" / name).write_bytes(b"x")
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild",
|
||||
@@ -903,6 +1149,59 @@ def test_rebuild_of_an_unknown_job_is_404(tmp_path):
|
||||
assert client.post("/api/jobs/nope/rebuild", data=rebuild_form()).status_code == 404
|
||||
|
||||
|
||||
def test_rebuild_full_forces_every_stage_from_the_first(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, capture_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild-full").json()["id"]
|
||||
wait_status(client, child, "done")
|
||||
assert seen[1]["force_from"] == STAGE_ORDER[0]
|
||||
assert seen[1]["inp"] == seen[0]["inp"]
|
||||
|
||||
|
||||
def test_rebuild_full_creates_a_linked_job(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
resp = client.post(f"/api/jobs/{parent}/rebuild-full")
|
||||
assert resp.status_code == 201
|
||||
child = resp.json()["id"]
|
||||
assert child != parent
|
||||
job = wait_status(client, child, "done")
|
||||
assert job["rebuilt_from"] == parent
|
||||
|
||||
|
||||
def test_rebuild_full_logs_that_it_is_a_forced_rebuild(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild-full").json()["id"]
|
||||
job = wait_status(client, child, "done")
|
||||
rebuild_lines = [l for l in job["log"] if l.startswith("[rebuild]")]
|
||||
assert len(rebuild_lines) == 1
|
||||
assert STAGE_ORDER[0] in rebuild_lines[0]
|
||||
|
||||
|
||||
def test_rebuild_full_of_an_unknown_job_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
assert client.post("/api/jobs/nope/rebuild-full").status_code == 404
|
||||
|
||||
|
||||
def test_rebuild_full_of_a_running_job_is_409(tmp_path):
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(inp, cfg, log=print, force_from=None, stages=None):
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, blocking)
|
||||
parent = client.post("/api/jobs", data=rebuild_form()).json()["id"]
|
||||
try:
|
||||
wait_status(client, parent, "running")
|
||||
assert client.post(f"/api/jobs/{parent}/rebuild-full").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, parent, "done")
|
||||
|
||||
|
||||
def slug_runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
out = cfg.output_dir / song_slug(inp)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
@@ -951,6 +1250,76 @@ def test_rebuild_of_a_failed_parent_removes_nothing(tmp_path):
|
||||
assert (cfg.output_dir / "A - T" / "keep.txt").exists()
|
||||
|
||||
|
||||
def test_delete_removes_work_and_output_and_the_job(tmp_path):
|
||||
client, cfg = make_client(tmp_path, slug_runner)
|
||||
job_id = make_done_job(client)
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True, exist_ok=True)
|
||||
(cfg.work_dir / "A - T" / "audio.wav").write_bytes(b"x")
|
||||
assert (cfg.output_dir / "A - T").exists()
|
||||
|
||||
resp = client.delete(f"/api/jobs/{job_id}")
|
||||
|
||||
assert resp.status_code == 204
|
||||
assert not (cfg.work_dir / "A - T").exists()
|
||||
assert not (cfg.output_dir / "A - T").exists()
|
||||
assert client.get(f"/api/jobs/{job_id}").status_code == 404
|
||||
assert job_id not in [j["id"] for j in client.get("/api/jobs").json()]
|
||||
|
||||
|
||||
def test_delete_tolerates_a_work_dir_that_is_already_gone(tmp_path):
|
||||
client, cfg = make_client(tmp_path, slug_runner)
|
||||
job_id = make_done_job(client)
|
||||
shutil.rmtree(cfg.work_dir / "A - T")
|
||||
resp = client.delete(f"/api/jobs/{job_id}")
|
||||
assert resp.status_code == 204
|
||||
|
||||
|
||||
def test_delete_of_an_unknown_job_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
assert client.delete("/api/jobs/nope").status_code == 404
|
||||
|
||||
|
||||
def test_delete_of_a_running_job_is_409(tmp_path):
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(inp, cfg, log=print, force_from=None, stages=None):
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, blocking)
|
||||
parent = client.post("/api/jobs", data=rebuild_form()).json()["id"]
|
||||
try:
|
||||
wait_status(client, parent, "running")
|
||||
assert client.delete(f"/api/jobs/{parent}").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, parent, "done")
|
||||
assert client.get(f"/api/jobs/{parent}").status_code == 200
|
||||
|
||||
|
||||
def test_delete_of_a_job_with_an_active_rebuild_is_409(tmp_path):
|
||||
"""A done job's slug must not be deleted out from under a rebuild in flight."""
|
||||
release = threading.Event()
|
||||
calls = []
|
||||
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
calls.append(1)
|
||||
if len(calls) > 1:
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, runner)
|
||||
parent = make_done_job(client)
|
||||
child = client.post(f"/api/jobs/{parent}/rebuild-full").json()["id"]
|
||||
try:
|
||||
wait_status(client, child, "running")
|
||||
assert client.delete(f"/api/jobs/{parent}").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, child, "done")
|
||||
assert client.get(f"/api/jobs/{parent}").status_code == 200
|
||||
|
||||
|
||||
def test_a_plain_job_never_removes_anything(tmp_path):
|
||||
client, cfg = make_client(tmp_path, slug_runner)
|
||||
make_done_job(client)
|
||||
@@ -958,3 +1327,422 @@ def test_a_plain_job_never_removes_anything(tmp_path):
|
||||
stray.mkdir(parents=True)
|
||||
make_done_job(client, title="T2")
|
||||
assert stray.exists()
|
||||
|
||||
|
||||
def test_discover_jobs_ignores_a_work_dir_without_a_sidecar(tmp_path):
|
||||
from songtrace.web import _discover_jobs
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
(cfg.work_dir / "A - T").mkdir(parents=True)
|
||||
assert _discover_jobs(cfg) == []
|
||||
|
||||
|
||||
def test_discover_jobs_reconstructs_a_completed_song(tmp_path):
|
||||
from songtrace.web import _discover_jobs
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
client = TestClient(create_app(cfg, runner=slug_runner))
|
||||
make_done_job(client)
|
||||
|
||||
discovered = _discover_jobs(cfg)
|
||||
|
||||
assert len(discovered) == 1
|
||||
job = discovered[0]
|
||||
assert job.title == "T" and job.artist == "A" and job.status == "done"
|
||||
assert job.output_dir == str(cfg.output_dir / "A - T")
|
||||
assert job.error is None
|
||||
|
||||
|
||||
def test_discover_jobs_reconstructs_a_song_whose_output_is_missing_as_failed(tmp_path):
|
||||
from songtrace.web import _discover_jobs
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
client = TestClient(create_app(cfg, runner=slug_runner))
|
||||
make_done_job(client)
|
||||
shutil.rmtree(cfg.output_dir / "A - T")
|
||||
|
||||
discovered = _discover_jobs(cfg)
|
||||
|
||||
assert len(discovered) == 1
|
||||
assert discovered[0].status == "failed"
|
||||
assert discovered[0].error == "interrupted before completion"
|
||||
assert discovered[0].output_dir is None
|
||||
|
||||
|
||||
def test_discovered_jobs_are_visible_from_a_fresh_app(tmp_path):
|
||||
cfg = Config(whisper_url="u", work_dir=tmp_path / "work", output_dir=tmp_path / "out")
|
||||
client1 = TestClient(create_app(cfg, runner=slug_runner))
|
||||
make_done_job(client1)
|
||||
|
||||
client2 = TestClient(create_app(cfg, runner=slug_runner))
|
||||
jobs = client2.get("/api/jobs").json()
|
||||
assert len(jobs) == 1
|
||||
assert jobs[0]["title"] == "T" and jobs[0]["status"] == "done"
|
||||
|
||||
|
||||
def review_runner_factory(notes, dropped_lines=None, missing_spans=None, warnings=None,
|
||||
words=None):
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
work = cfg.work_dir / song_slug(inp)
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
(work / "notes_sec.json").write_text(json.dumps({
|
||||
"notes": notes, "breaks": [], "warnings": warnings or []}))
|
||||
if dropped_lines is not None or missing_spans is not None:
|
||||
(work / "align_meta.json").write_text(json.dumps({
|
||||
"dropped_lines": dropped_lines or [],
|
||||
"missing_spans": missing_spans or []}))
|
||||
if words is not None:
|
||||
(work / "alignment.json").write_text(json.dumps(words))
|
||||
out = cfg.output_dir / song_slug(inp)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
return PipelineResult(output_dir=out, warnings=warnings or [])
|
||||
return runner
|
||||
|
||||
|
||||
def test_review_endpoint_groups_notes_by_line(tmp_path):
|
||||
notes = [
|
||||
{"start": 1.0, "duration": 0.4, "pitch": 0, "text": "hi",
|
||||
"low_confidence": False, "line_index": 0},
|
||||
{"start": 5.0, "duration": 0.3, "pitch": 2, "text": "there",
|
||||
"low_confidence": True, "line_index": 1},
|
||||
]
|
||||
client, _ = make_client(tmp_path, review_runner_factory(
|
||||
notes, dropped_lines=[2], missing_spans=[[8.0, 9.5]]))
|
||||
resp = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "hi\nthere\ngone",
|
||||
"youtube_url": "http://y"})
|
||||
job_id = resp.json()["id"]
|
||||
wait_status(client, job_id, "done")
|
||||
body = client.get(f"/api/jobs/{job_id}/review").json()
|
||||
assert body["lines"] == [
|
||||
{"index": 0, "text": "hi", "dropped": False,
|
||||
"notes": [{"start": 1.0, "end": 1.4, "pitch": 0, "low_confidence": False}],
|
||||
"words": []},
|
||||
{"index": 1, "text": "there", "dropped": False,
|
||||
"notes": [{"start": 5.0, "end": 5.3, "pitch": 2, "low_confidence": True}],
|
||||
"words": []},
|
||||
{"index": 2, "text": "gone", "dropped": True, "notes": [], "words": []},
|
||||
]
|
||||
assert body["missing_spans"] == [{"start": 8.0, "end": 9.5}]
|
||||
|
||||
|
||||
def test_review_endpoint_includes_per_word_timing(tmp_path):
|
||||
notes = [
|
||||
{"start": 1.0, "duration": 0.4, "pitch": 0, "text": "hi",
|
||||
"low_confidence": False, "line_index": 0},
|
||||
{"start": 5.0, "duration": 0.3, "pitch": 2, "text": "there",
|
||||
"low_confidence": True, "line_index": 1},
|
||||
]
|
||||
words = [
|
||||
{"word": "hi", "start": 1.0, "end": 1.4, "score": 0.9, "line_index": 0},
|
||||
{"word": "there", "start": 5.0, "end": 5.3, "score": 0.9, "line_index": 1},
|
||||
]
|
||||
client, _ = make_client(tmp_path, review_runner_factory(
|
||||
notes, dropped_lines=[2], missing_spans=[[8.0, 9.5]], words=words))
|
||||
resp = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "hi\nthere\ngone",
|
||||
"youtube_url": "http://y"})
|
||||
job_id = resp.json()["id"]
|
||||
wait_status(client, job_id, "done")
|
||||
body = client.get(f"/api/jobs/{job_id}/review").json()
|
||||
assert body["lines"][0]["words"] == [{"text": "hi", "start": 1.0, "end": 1.4}]
|
||||
assert body["lines"][1]["words"] == [{"text": "there", "start": 5.0, "end": 5.3}]
|
||||
assert body["lines"][2]["dropped"] is True
|
||||
assert body["lines"][2]["words"] == []
|
||||
|
||||
|
||||
def test_review_of_a_failed_job_is_404(tmp_path):
|
||||
def bad_runner(*a, **kw):
|
||||
raise PipelineError("boom")
|
||||
client, _ = make_client(tmp_path, bad_runner)
|
||||
resp = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"})
|
||||
job_id = resp.json()["id"]
|
||||
wait_status(client, job_id, "failed")
|
||||
assert client.get(f"/api/jobs/{job_id}/review").status_code == 404
|
||||
|
||||
|
||||
def test_review_of_a_running_job_is_409(tmp_path):
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(inp, cfg, log=print, force_from=None, stages=None):
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, blocking)
|
||||
resp = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"})
|
||||
job_id = resp.json()["id"]
|
||||
try:
|
||||
wait_status(client, job_id, "running")
|
||||
assert client.get(f"/api/jobs/{job_id}/review").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, job_id, "done")
|
||||
|
||||
|
||||
def test_review_of_an_unknown_job_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
assert client.get("/api/jobs/nope/review").status_code == 404
|
||||
|
||||
|
||||
def test_review_missing_when_pipeline_never_wrote_notes(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/review").status_code == 404
|
||||
|
||||
|
||||
# --- review playback -------------------------------------------------------
|
||||
|
||||
MEDIA_FILES = {"A - T.mp3": b"ID3audio", "A - T.mp4": b"\0\0\0\x18ftypvideo",
|
||||
"A - T.jpg": b"\xff\xd8cover"}
|
||||
MEDIA_HEADERS = [("TITLE", "T"), ("ARTIST", "A"), ("MP3", "A - T.mp3"),
|
||||
("COVER", "A - T.jpg"), ("VIDEO", "A - T.mp4")]
|
||||
|
||||
|
||||
def media_runner_factory(files=None, headers=None, write_txt=True):
|
||||
"""A run that leaves behind a song folder, the way package does."""
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
slug = song_slug(inp)
|
||||
out = cfg.output_dir / slug
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
for name, body in (MEDIA_FILES if files is None else files).items():
|
||||
(out / name).write_bytes(body)
|
||||
if write_txt:
|
||||
rows = [f"#{k}:{v}" for k, v in
|
||||
(MEDIA_HEADERS if headers is None else headers)]
|
||||
(out / f"{slug}.txt").write_text("\n".join(rows) + "\n: 0 4 0 la\nE\n",
|
||||
encoding="utf-8")
|
||||
return PipelineResult(output_dir=out)
|
||||
return runner
|
||||
|
||||
|
||||
def test_media_serves_the_files_the_song_txt_names(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
job_id = make_done_job(client)
|
||||
for kind, name in (("audio", "A - T.mp3"), ("video", "A - T.mp4"),
|
||||
("cover", "A - T.jpg")):
|
||||
resp = client.get(f"/api/jobs/{job_id}/media/{kind}")
|
||||
assert resp.status_code == 200, kind
|
||||
assert resp.content == MEDIA_FILES[name], kind
|
||||
|
||||
|
||||
def test_media_serves_whatever_the_header_was_rewritten_to(tmp_path):
|
||||
"""package transcodes to .mp4 and rewrites #VIDEO; the extension is not fixed."""
|
||||
files = {**MEDIA_FILES, "A - T.webm": b"transcoded"}
|
||||
headers = [(k, "A - T.webm" if k == "VIDEO" else v) for k, v in MEDIA_HEADERS]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(files, headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/video").content == b"transcoded"
|
||||
|
||||
|
||||
def test_media_supports_range_requests(tmp_path):
|
||||
"""Without 206 the browser cannot seek, which is the point of the player."""
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
job_id = make_done_job(client)
|
||||
resp = client.get(f"/api/jobs/{job_id}/media/video", headers={"Range": "bytes=0-3"})
|
||||
assert resp.status_code == 206
|
||||
assert resp.headers["content-range"] == f"bytes 0-3/{len(MEDIA_FILES['A - T.mp4'])}"
|
||||
assert resp.content == MEDIA_FILES["A - T.mp4"][:4]
|
||||
|
||||
|
||||
def test_media_of_an_unknown_kind_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/instrumental").status_code == 404
|
||||
|
||||
|
||||
def test_media_refuses_a_header_pointing_outside_the_song_folder(tmp_path):
|
||||
secret = tmp_path / "secret.txt"
|
||||
secret.write_text("not yours")
|
||||
headers = [(k, "../../secret.txt" if k == "MP3" else v) for k, v in MEDIA_HEADERS]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(headers=headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_refuses_an_absolute_header_path(tmp_path):
|
||||
secret = tmp_path / "secret.txt"
|
||||
secret.write_text("not yours")
|
||||
headers = [(k, str(secret) if k == "MP3" else v) for k, v in MEDIA_HEADERS]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(headers=headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_404s_when_the_named_file_is_missing(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory(files={}))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_404s_when_the_header_was_dropped(tmp_path):
|
||||
"""package strips #COVER when there is no usable cover image."""
|
||||
headers = [(k, v) for k, v in MEDIA_HEADERS if k != "COVER"]
|
||||
client, _ = make_client(tmp_path, media_runner_factory(headers=headers))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/cover").status_code == 404
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 200
|
||||
|
||||
|
||||
def test_media_404s_when_there_is_no_song_txt(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory(write_txt=False))
|
||||
job_id = make_done_job(client)
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_of_a_failed_job_is_404(tmp_path):
|
||||
def bad_runner(*a, **kw):
|
||||
raise PipelineError("boom")
|
||||
client, _ = make_client(tmp_path, bad_runner)
|
||||
job_id = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"}).json()["id"]
|
||||
wait_status(client, job_id, "failed")
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_media_of_a_running_job_is_409(tmp_path):
|
||||
release = threading.Event()
|
||||
|
||||
def blocking(inp, cfg, log=print, force_from=None, stages=None):
|
||||
release.wait(5)
|
||||
return PipelineResult(output_dir=cfg.output_dir / "X")
|
||||
|
||||
client, _ = make_client(tmp_path, blocking)
|
||||
job_id = client.post("/api/jobs", data={
|
||||
"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"}).json()["id"]
|
||||
try:
|
||||
wait_status(client, job_id, "running")
|
||||
assert client.get(f"/api/jobs/{job_id}/media/audio").status_code == 409
|
||||
finally:
|
||||
release.set()
|
||||
wait_status(client, job_id, "done")
|
||||
|
||||
|
||||
def test_media_of_an_unknown_job_is_404(tmp_path):
|
||||
client, _ = make_client(tmp_path, media_runner_factory())
|
||||
assert client.get("/api/jobs/nope/media/audio").status_code == 404
|
||||
|
||||
|
||||
def test_review_reports_which_media_the_player_can_load(tmp_path):
|
||||
notes = [{"start": 1.0, "duration": 0.4, "pitch": 0, "text": "hi",
|
||||
"low_confidence": False, "line_index": 0}]
|
||||
review = review_runner_factory(notes)
|
||||
media = media_runner_factory(headers=[(k, v) for k, v in MEDIA_HEADERS
|
||||
if k != "COVER"])
|
||||
|
||||
def runner(inp, cfg, **kw):
|
||||
media(inp, cfg, **kw)
|
||||
return review(inp, cfg, **kw)
|
||||
|
||||
client, _ = make_client(tmp_path, runner)
|
||||
job_id = make_done_job(client, lyrics="hi")
|
||||
body = client.get(f"/api/jobs/{job_id}/review").json()
|
||||
assert body["media"] == {"audio": True, "video": True, "cover": False}
|
||||
|
||||
|
||||
# --- restarting a rebuild from a chosen stage ------------------------------
|
||||
|
||||
def force_from_runner_factory(seen):
|
||||
def runner(inp, cfg, log=print, force_from=None, stages=None):
|
||||
seen.append(force_from)
|
||||
out = cfg.output_dir / song_slug(inp)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
return PipelineResult(output_dir=out)
|
||||
return runner
|
||||
|
||||
|
||||
def rebuild_with(client, job_id, **extra):
|
||||
data = {"title": "T", "artist": "A", "lyrics": "x", "youtube_url": "http://y"}
|
||||
data.update(extra)
|
||||
return client.post(f"/api/jobs/{job_id}/rebuild", data=data)
|
||||
|
||||
|
||||
def test_rebuild_without_edits_restarts_from_the_stage_asked_for(tmp_path):
|
||||
"""The point of the control: a change in songtrace invalidates no field."""
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, force_from="package")
|
||||
assert resp.status_code == 201
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "package"
|
||||
|
||||
|
||||
def test_rebuild_without_a_stage_still_plans_from_the_edit(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, lyrics="different words")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "align"
|
||||
|
||||
|
||||
def test_an_edit_wins_when_it_invalidates_more_than_the_stage_asked_for(tmp_path):
|
||||
"""A choice may widen what is redone, never narrow it below correctness."""
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, lyrics="different words",
|
||||
force_from="package")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "align"
|
||||
|
||||
|
||||
def test_the_stage_asked_for_wins_when_it_is_earlier_than_the_edit(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, title="Renamed", force_from="separate")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == "separate"
|
||||
|
||||
|
||||
def test_rebuild_rejects_a_stage_that_is_not_in_the_pipeline(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, force_from="deechoing")
|
||||
assert resp.status_code == 400
|
||||
assert "deechoing" in resp.json()["detail"]
|
||||
|
||||
|
||||
def test_a_blank_stage_is_the_same_as_not_asking(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = rebuild_with(client, parent, force_from="")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] is None
|
||||
|
||||
|
||||
def test_rebuild_from_scratch_still_restarts_at_the_first_stage(tmp_path):
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client)
|
||||
resp = client.post(f"/api/jobs/{parent}/rebuild-full")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] == STAGE_ORDER[0]
|
||||
|
||||
|
||||
def test_the_restart_control_offers_every_stage():
|
||||
html = _index_html()
|
||||
assert 'name="force_from"' in html
|
||||
stages = _fn_body(html, "loadStages")
|
||||
assert "force_from" in stages and "STAGES.map" in stages
|
||||
mode = _fn_body(html, "setFormMode")
|
||||
assert "restart-field" in mode, "the control is shown outside rebuild mode"
|
||||
|
||||
|
||||
def test_a_textareas_crlf_does_not_read_as_a_lyrics_edit(tmp_path):
|
||||
"""Browsers submit textareas with CRLF; the stored lyrics use LF."""
|
||||
seen = []
|
||||
client, _ = make_client(tmp_path, force_from_runner_factory(seen))
|
||||
parent = make_done_job(client, lyrics="one\ntwo")
|
||||
resp = rebuild_with(client, parent, lyrics="one\r\ntwo")
|
||||
wait_status(client, resp.json()["id"], "done")
|
||||
assert seen[-1] is None, "the rebuild redid the alignment for nothing"
|
||||
body = client.get(f"/api/jobs/{resp.json()['id']}/input").json()
|
||||
assert body["lyrics"] == "one\ntwo"
|
||||
|
||||
|
||||
def test_submitting_crlf_lyrics_stores_them_with_lf(tmp_path):
|
||||
client, _ = make_client(tmp_path, ok_runner)
|
||||
job_id = make_done_job(client, lyrics="one\r\ntwo")
|
||||
assert client.get(f"/api/jobs/{job_id}/input").json()["lyrics"] == "one\ntwo"
|
||||
|
||||
+24
-2
@@ -1,7 +1,7 @@
|
||||
import httpx
|
||||
from pathlib import Path
|
||||
from karaoke_maker.config import Config
|
||||
from karaoke_maker.whisper_client import TranscriptSegment, transcribe
|
||||
from songtrace.config import Config
|
||||
from songtrace.whisper_client import TranscriptSegment, transcribe
|
||||
|
||||
def test_transcribe_parses_segments(tmp_path):
|
||||
audio = tmp_path / "v.wav"
|
||||
@@ -36,3 +36,25 @@ def test_transcribe_maps_language_to_iso_code(tmp_path):
|
||||
transcribe(cfg, audio, language="Japanese", client=client)
|
||||
assert 'name="language"' in seen["body"]
|
||||
assert "\r\n\r\nja\r\n" in seen["body"]
|
||||
|
||||
|
||||
def test_transcribe_asr_webservice_hits_fixed_asr_endpoint(tmp_path):
|
||||
audio = tmp_path / "v.wav"
|
||||
audio.write_bytes(b"RIFF")
|
||||
seen = {}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen["url"] = str(request.url)
|
||||
seen["body"] = request.read().decode("utf-8", errors="replace")
|
||||
return httpx.Response(200, json={
|
||||
"text": "hello world",
|
||||
"segments": [{"start": 0.5, "end": 2.0, "text": " hello world"}],
|
||||
})
|
||||
|
||||
client = httpx.Client(transport=httpx.MockTransport(handler))
|
||||
cfg = Config(whisper_url="http://w:9000", whisper_api="asr-webservice")
|
||||
segs = transcribe(cfg, audio, language="Japanese", client=client)
|
||||
assert segs == [TranscriptSegment(start=0.5, end=2.0, text="hello world")]
|
||||
assert seen["url"] == "http://w:9000/asr?output=json&language=ja"
|
||||
assert 'name="audio_file"' in seen["body"]
|
||||
assert 'name="model"' not in seen["body"]
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
import os
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from songtrace import ytdlp
|
||||
from songtrace.config import Config
|
||||
from songtrace.ytdlp import YtDlp, YtDlpError
|
||||
|
||||
|
||||
def fake_package(path, version="2026.09.01", body=""):
|
||||
"""A stand-in yt_dlp that answers --version and otherwise runs `body`."""
|
||||
pkg = path / "yt_dlp"
|
||||
pkg.mkdir(parents=True)
|
||||
(pkg / "__init__.py").touch()
|
||||
(pkg / "__main__.py").write_text(
|
||||
"import sys\n"
|
||||
f"if '--version' in sys.argv: print({version!r}); sys.exit()\n"
|
||||
+ body)
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cache(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "cache"))
|
||||
monkeypatch.setattr(ytdlp, "bundled", lambda: YtDlp("2026.8.19"))
|
||||
return ytdlp.cache_dir()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pypi(monkeypatch):
|
||||
"""Stubs PyPI and pip: `latest` is the newest release, `installs` logs pip."""
|
||||
state = {"latest": "2026.9.1", "installs": []}
|
||||
monkeypatch.setattr(ytdlp, "_latest_release", lambda: state["latest"])
|
||||
def pip_install(target, version):
|
||||
state["installs"].append(version)
|
||||
fake_package(target, version)
|
||||
monkeypatch.setattr(ytdlp, "_pip_install", pip_install)
|
||||
return state
|
||||
|
||||
|
||||
def cfg(version="auto"):
|
||||
return Config(whisper_url="http://x/v1", ytdlp_version=version)
|
||||
|
||||
|
||||
def test_run_uses_the_install_on_its_path_and_logs_its_output(tmp_path):
|
||||
path = fake_package(tmp_path, body="print('[youtube] hello', sys.argv[1:])\n")
|
||||
logged = []
|
||||
YtDlp("2026.09.01", path).run(["--", "https://yt/x"], logged.append)
|
||||
assert logged == ["[youtube] hello ['--ignore-config', '--', 'https://yt/x']"]
|
||||
|
||||
|
||||
def test_run_raises_the_error_line(tmp_path):
|
||||
path = fake_package(tmp_path, body=(
|
||||
"print('ERROR: HTTP Error 403: Forbidden', file=sys.stderr)\n"
|
||||
"print('trailing noise')\nsys.exit(1)\n"))
|
||||
with pytest.raises(YtDlpError, match="^ERROR: HTTP Error 403"):
|
||||
YtDlp("2026.09.01", path).run([], lambda m: None)
|
||||
|
||||
|
||||
def test_run_flags_a_usage_error_as_incompatible(tmp_path):
|
||||
path = fake_package(tmp_path, body=(
|
||||
"print('error: no such option: --retries', file=sys.stderr)\n"
|
||||
"sys.exit(2)\n"))
|
||||
with pytest.raises(YtDlpError, match="2026.09.01 rejected.*--retries"):
|
||||
YtDlp("2026.09.01", path).run([], lambda m: None)
|
||||
|
||||
|
||||
def test_auto_installs_a_newer_release_and_tries_it_first(cache, pypi):
|
||||
found = ytdlp.candidates(cfg(), lambda m: None)
|
||||
assert [y.version for y in found] == ["2026.9.1", "2026.8.19"]
|
||||
assert found[0].path == cache / "2026.9.1"
|
||||
assert not [p for p in cache.iterdir() if p.name.startswith(".tmp-")]
|
||||
|
||||
|
||||
def test_auto_checks_pypi_once_a_day(cache, pypi):
|
||||
ytdlp.candidates(cfg(), lambda m: None)
|
||||
pypi["latest"] = "2026.10.1"
|
||||
found = ytdlp.candidates(cfg(), lambda m: None)
|
||||
assert found[0].version == "2026.9.1"
|
||||
found = ytdlp.candidates(cfg(), lambda m: None, refresh=True)
|
||||
assert found[0].version == "2026.10.1"
|
||||
assert not (cache / "2026.9.1").exists()
|
||||
old = time.time() - ytdlp.CHECK_INTERVAL - 1
|
||||
os.utime(cache / ".checked", (old, old))
|
||||
pypi["latest"] = "2026.11.1"
|
||||
assert ytdlp.candidates(cfg(), lambda m: None)[0].version == "2026.11.1"
|
||||
|
||||
|
||||
def test_auto_uses_only_bundled_when_it_is_newest(cache, pypi):
|
||||
pypi["latest"] = "2026.8.19"
|
||||
assert [y.version for y in ytdlp.candidates(cfg(), lambda m: None)] == [
|
||||
"2026.8.19"]
|
||||
assert pypi["installs"] == []
|
||||
|
||||
|
||||
def test_auto_survives_pypi_being_unreachable(cache, pypi, monkeypatch):
|
||||
ytdlp.candidates(cfg(), lambda m: None)
|
||||
def down():
|
||||
raise OSError("network unreachable")
|
||||
monkeypatch.setattr(ytdlp, "_latest_release", down)
|
||||
logged = []
|
||||
found = ytdlp.candidates(cfg(), logged.append, refresh=True)
|
||||
assert [y.version for y in found] == ["2026.9.1", "2026.8.19"]
|
||||
assert any("network unreachable" in m for m in logged)
|
||||
|
||||
|
||||
def test_an_install_that_fails_its_check_is_discarded(cache, pypi, monkeypatch):
|
||||
monkeypatch.setattr(ytdlp, "_pip_install",
|
||||
lambda target, v: fake_package(target, "2026.1.1"))
|
||||
logged = []
|
||||
assert [y.version for y in ytdlp.candidates(cfg(), logged.append)] == [
|
||||
"2026.8.19"]
|
||||
assert any("failed its check" in m for m in logged)
|
||||
assert list(cache.iterdir()) == [cache / ".checked"]
|
||||
|
||||
|
||||
def test_bundled_never_touches_pypi(cache, pypi):
|
||||
pypi["latest"] = None
|
||||
assert [y.version for y in ytdlp.candidates(cfg("bundled"), print)] == [
|
||||
"2026.8.19"]
|
||||
assert not cache.exists()
|
||||
|
||||
|
||||
def test_pin_installs_that_release_even_if_older(cache, pypi):
|
||||
found = ytdlp.candidates(cfg("2026.07.01"), lambda m: None)
|
||||
assert [y.version for y in found] == ["2026.07.01", "2026.8.19"]
|
||||
assert pypi["installs"] == ["2026.07.01"]
|
||||
|
||||
|
||||
def test_pin_matching_bundled_installs_nothing(cache, pypi):
|
||||
found = ytdlp.candidates(cfg("2026.08.19"), lambda m: None)
|
||||
assert [y.version for y in found] == ["2026.8.19"]
|
||||
assert pypi["installs"] == []
|
||||
|
||||
|
||||
def test_pin_must_be_a_release_number(cache, pypi):
|
||||
with pytest.raises(YtDlpError, match="ytdlp_version"):
|
||||
ytdlp.candidates(cfg("latest; rm -rf /"), lambda m: None)
|
||||
Reference in New Issue
Block a user