Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lyrics: Fetch lyrics directly from Tekstowo #5457

Merged
merged 2 commits into from
Oct 12, 2024
Merged

Lyrics: Fetch lyrics directly from Tekstowo #5457

merged 2 commits into from
Oct 12, 2024

Conversation

snejus
Copy link
Member

@snejus snejus commented Oct 12, 2024 •
edited
Loading

Summary

Fixes #5456.

This pull request updates the tekstowo backend in the lyrics plugin to fetch lyrics directly from the Tekstowo.pl website. Recent updates to their website made the previous search-based approach unworkable.

Changes

  1. Refactor Backend Classes:

    • Introduced a new DirectBackend class for backends that fetch lyrics directly.
    • Updated MusiXmatch and Tekstowo classes to inherit from DirectBackend.

  2. Encoding and URL Building:

    • Added encode and build_url methods to DirectBackend for URL encoding and construction.
    • Replaced our custom encoding functionality with unidecode.

  3. Tekstowo Backend:

    • Added encoding logic, which converts artist and title to the format used by tekstowo.
    • Removed the search functionality (parse_search_results), its test and related search html files.
    • Updated artist and title checks from extract_lyrics.

Test

$ beet -vv lyrics artist:kendrick -f
user configuration: /home/sarunas/.config/beets/config.yaml
data directory: /home/sarunas/.config/beets
plugin paths: 
inline: adding item field label_or_albumartist
inline: adding item field singleton_track_artist
inline: adding item field track_artist
inline: adding item field album_name
inline: adding item field track_identification
inline: adding item field withdrawn
inline: adding album field multiple_artists
fetchart: google: Disabling art source due to missing key
Sending event: pluginload
library database: /home/sarunas/.music/beets/library.db
library directory: /run/media/sarunas/music/Music
Sending event: library_opened
Parsed query: AndQuery([SubstringQuery('artist', 'kendrick', fast=True)])
Parsed sort: NullSort()
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 36218 | 2012 / Swimming Pools (Drank): Kendrick Lamar - Swimming Pools (Drank)
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43230 | 2015 / To Pimp a Butterfly: Kendrick Lamar - For Free? (interlude)
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43231 | 2015 / To Pimp a Butterfly: Kendrick Lamar - King Kunta
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43234 | 2015 / To Pimp a Butterfly: Kendrick Lamar - u
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43235 | 2015 / To Pimp a Butterfly: Kendrick Lamar - Alright
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43236 | 2015 / To Pimp a Butterfly: Kendrick Lamar - For Sale? (interlude)
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43237 | 2015 / To Pimp a Butterfly: Kendrick Lamar - Momma
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43238 | 2015 / To Pimp a Butterfly: Kendrick Lamar - Hood Politics
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43241 | 2015 / To Pimp a Butterfly: Kendrick Lamar - The Blacker the Berry
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43242 | 2015 / To Pimp a Butterfly: Kendrick Lamar - You Ain't Gotta Lie (Momma Said)
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43243 | 2015 / To Pimp a Butterfly: Kendrick Lamar - i
Sending event: database_change
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43244 | 2015 / To Pimp a Butterfly: Kendrick Lamar - Mortal Man
Sending event: database_change
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,kendrick_lamar_ft__bilal__anna_wise____snoop_dogg,institutionalized.html (404)
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43232 | 2015 / To Pimp a Butterfly: Kendrick Lamar ft. Bilal, Anna Wise, & Snoop Dogg - Institutionalized
Sending event: database_change
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,kendrick_lamar_ft__bilal__anna_wise____thundercat,these_walls.html (404)
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43233 | 2015 / To Pimp a Butterfly: Kendrick Lamar ft. Bilal, Anna Wise, & Thundercat - These Walls
Sending event: database_change
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,kendrick_lamar_ft__george_clinton___thundercat,wesley_s_theory.html (404)
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43229 | 2015 / To Pimp a Butterfly: Kendrick Lamar ft. George Clinton & Thundercat - Wesley's Theory
Sending event: database_change
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,kendrick_lamar_ft__james_fauntleroy___ronald_isley,how_much_a_dollar_cost.html (404)
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,kendrick_lamar,how_much_a_dollar_cost.html (404)
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,lamar__kendrick_ft__fauntleroy__james___isley__ronald,how_much_a_dollar_cost.html (404)
lyrics: lyrics not found: 43239 | 2015 / To Pimp a Butterfly: Kendrick Lamar ft. James Fauntleroy & Ronald Isley - How Much a Dollar Cost
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,kendrick_lamar_ft__rapsody,complexion__a_zulu_love_.html (404)
lyrics: failed to fetch: https://www.tekstowo.pl/piosenka,kendrick_lamar_ft__rapsody,complexion.html (404)
lyrics: got lyrics from backend: Tekstowo
lyrics: fetched lyrics: 43240 | 2015 / To Pimp a Butterfly: Kendrick Lamar ft. Rapsody - Complexion (A Zulu Love)
Sending event: database_change
Sending event: cli_exit
mpdupdate: Updating MPD database...
mpdupdate: Database updated.

Compare this to the output I reported under #5456. All lyrics were found except for:

  • Kendrick Lamar ft. James Fauntleroy & Ronald Isley - How Much a Dollar Cost

I found them on Tekstowo manually where they are titled slightly differently:

  • Kendrick Lamar - How Much A Dollar Cost feat. James Fauntleroy & Ronald Isley.

I do not think there's an easy way around this though, unfortunately.

This nice upside is that this is now much faster than before!

To Do

  • Documentation. (If you've added a new command-line flag, for example, find the appropriate page under docs/ to describe it.)
  • Changelog. (Add an entry to docs/changelog.rst to the bottom of one of the lists near the top of the document.)
  • Tests. (Very much encouraged but not strictly required.)

@snejus
Create DirectBackend interface for MusiXmatch and Tekstowo
9d2b34d 
And replace some of the encoding logic by unidecode.
@snejus snejus self-assigned this Oct 12, 2024
@snejus
Update Tekstowo backend to fetch lyrics directly
d3955ba 
- Refactored Tekstowo backend to fetch lyrics directly from song pages.
- Added `encode` method to convert artist and title to their URL format,
  where non-alphanumeric characters are replaced with underscores.
- Removed the now redundant search functionality and associated tests.
- Simplified `extract_lyrics` method to directly parse lyrics without
  any checks.
@bal-e
Copy link
Member

bal-e commented Oct 12, 2024

That's an unfortunate change in Tekstowo. However, the direct lookup seems to be working well enough that it's not an issue in most cases, and a speed improvement sounds great.

bal-e
bal-e approved these changes Oct 12, 2024
View reviewed changes
Copy link
Member

@bal-e bal-e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! It's unfortunate that we lost functionality, but I'm happy we're trimming the codebase.

@snejus snejus merged commit 65d78cb into master Oct 12, 2024
15 checks passed
@snejus snejus deleted the lyrics-fix-tekstowo branch October 12, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bal-e bal-e bal-e approved these changes

@ybnd ybnd Awaiting requested review from ybnd

@Serene-Arc Serene-Arc Awaiting requested review from Serene-Arc

Assignees

@snejus snejus

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Tekstowo backend does not return lyrics any more
2 participants
@snejus @bal-e