lastgenre: replace built-in genre list with improved hierarchy

[spaceage64]

Description

The current genre list is alright but not very extensive, and also contains some oddities.

Replaces genres-tree.yaml and genres.txt with a significantly expanded and restructured genre hierarchy.

• 2,984 'main' (whitelisted) genres, up from ~740
  • Includes some much needed additions like 'Vaporwave', 'Hyperpop', 'Phonk', 'Cloud Rap', 'Midwest Emo', etc.
  • Also includes very extensive entries for 'world music' (which is not itself a genre). E.g., 'Singeli' (African), 'Son Cubano' (Latin American), 'Mandopop' (Asian), 'Klezmer' (Jewish), etc.
  • Properly lists actual genres from non-Western regions under their real genre instead of just a region. E.g., the current hierarchy lists genres like 'blues' and 'classical', but then also treats 'african' and 'asian' as genres. In my hierarchy, such music will be listed under their proper genre, not just a region. E.g., 'K-Pop' will be listed under Pop → Asian Pop → K-Pop, not asian → east asian → k-pop.
  • Excludes some vague/nonsensical entries, e.g., 'acoustic music' (not a genre--can be any genre), 'cassette culture' (also not a genre), 'dementia' (???), etc.
• Many aliases supported (9,300+, up from 1546), will all refer to a standardised 'main' entry. E.g., 'hip-hop' will map to 'Hip Hop', 'dnb' will map to 'Drum & Bass'.
  • Also includes basic ascii aliases. E.g., both 'candomblé' and 'candomble' will map to 'Candomblé'.
• Much work was put into organising the tree correctly, meaning that a proper hierarchy of a genre can be traced. Some examples:
  1. Electronic → Electronic Dance Music → House → Hard House → UK Hard House → Scouse House → Pumping House → Hardbass
  2. Classical → Asian Classical → East Asian Classical → Korean Classical → Korean Court → Jeong-ak → Gagok
  3. Folk → Traditional Folk → European Folk → Balkan Folk → Greek Folk → Cretan Folk → Rizitika
• Proper Case genre names. Also supports proper case if other letters than the first one need capitalisation, e.g., 'IDM', 'HexD', etc. Or will lowercase if needed, e.g., 'Lo-fi' instead of 'Lo-Fi'.
  • Will need some work changing the current capitalisation logic to actually produce this as output, but at least we have a good source for proper capitalisation to work with now.

Sources used:

• RateYourMusic
• Wikipedia List of Music Genres
• Discogs
• AllMusic

Fixes #6628

To Do

• Documentation. The proposed hierarchy comes properly capitalised by default. I think ideally, when capitalisation is activated, it will use use the untouched terminology, and otherwise just lowercase everything. Currently the setup is different, just applying Proper Case to any lowercase default value. If that logic is altered, we will need new documentation, but as it is no new documentation is required.
• Changelog. Will add this once it's approved.
• Tests. I use this hierarchy myself.

I don't know if my PR is even appreciated, but a lack of extensive genre maintenance was bothering me personally. The default list seems to be based on some very old extraction of Wikipedia and isn't necessarily of high quality. So I put a lot of effort into building this for personal use and I thought I'd at least offer to share. Please let me know if anything more would be required from my side.

[snejus]

@spaceage64 the tests are failing. @JOJ0 I'll leave this one to you!

[spaceage64]

Thanks. Yes, I think the Proper Case is causing the tests to fail. In practice, it shouldn't cause issues though. I have made the list Proper Case so we no longer have to use very simplistic code to have beets do the Proper Case conversion. Please let me know if more input is required from my end.

[JOJ0]

Hi @spaceage64 , yes this contribution is definitely appreciated. I see you put a lot, and I mean A LOT of effort in it 🤩 , that is really awesome and I think we can and should use/ship a lot of it with beets!

But we can't simply accept it like it is! There are several issues I'm seeing and I have to warn you beforehand that it might be a lot of work to make this mergeable. If you are willing to work on it further and have some patience, together we might be able to put it to good use.

• You seem to use the canonicaliztaion feature as a "spelling correction", that is ok and there was no other way in the past. We are about to merge a PR that solves that problem differently and frankly - better and more maintainable. Have a look at the list of default aliases it will ship. This is only a minimal start I worked on my end but It might get improved with your spelling findings: https://github.com/beetbox/beets/blob/fa0d3c2f8aa2bff40138a9e9e099c7fb9fbf6dbe/beetsplug/lastgenre/aliases.yaml

• Since all the features available in the lastgenre plugin can be used in several ways I added a guide on what I suggest to use for certain use-cases and (also important) in what order to use them! It's a quick read: lastgenre: Genre spelling normalization (aliases) #6466 (comment)

• Regarding the correct spelling you put (a lot of) effort in, I have to think about how to use it, it might get more difficult than you think since the whole plugin currently works somewhat like: Make everything lowercase, regex match lower case, do anything and everything lower case because it's more simple code, and only finally automagically turn everything into auto-generated case that is not perfect at all but good enough) -> That is worth a separate PR/brainstorming/feature request!

• And finally there is the point of opinionation. We have to be very careful to not overnormalize things. If we ship something with beets it should handle very common cases and I found myself removing a lot of normalizations again in the default aliases list I'd like to ship. Here often "less is more", only if they work for me doesn't mean they work for everybody (nothing works for everybody, but might work for a lot of people at least, that could be a goal!)

So maybe it is best we wait until the normalization feature is merged (I think we are close to merge already) and then let's brainstorm on how to proceed here. What do you think?

[spaceage64]

Thanks @JOJ0, sounds fine to me. My list is very extensive, so it already kind of resolves the issue of matching aliases to a singular genre by just using many alternative terms, instead of using regex. I suppose regex can be used for some common cases, but in many others, aliases really are different from each other in ways that regex won't be able to capture. So for my personal use it actually works fine already this way, without the new regex logic. I think the best way to combine the two in the future is probably to remove any aliases that may also be matched using regex and keep the others.

As for spelling, I'd really advise against just using simple lower to Proper Case conversion. It's way too simplistic to cover outliers, let alone non-English ones. Perhaps it may simplify the code, but it lowers the quality of the output. If there would be interest in using my hierarchy, it would already come with proper spelling by default, so I think it would be a waste to discard that and use inferior logic instead. Besides, even when only using lowercase in the code, my hierarchy is setup so that each Proper Case entry has directly under it a lower case variant used for matching. But as these are aliases, the Proper Case variant (i.e., the parent) should be returned instead. Perhaps I'm overlooking something, but it seems to me the logic already functions correctly. The only thing that would need changing, is getting rid of the additional Proper Case conversion the code currently includes. This would be the logic: lowercase any terms > regex lookup against aliases (all lowercase as well) > return Proper Case parent > done, no additional (potentially incorrect) conversion required.

I agree the hierarchy is a product of my own personal ideas of what a proper structure should look like. I'd tried to include as much as possible:

• 'Normal' genres: Metal, Pop, Rock, etc.
• Regional genres: These will still be assigned to the proper 'normal' genres, but for any significant regional variants/scenes, I have made separate sub entries. E.g., Korean Classical, East Coast Hip Hop, etc.
• 'Audio Media', less standard non-music content, such as podcasts, speeches, educational material, radio dramas, etc.

Even for my own use case, I don't always want to use the genres suggested by the plugin, so I use a hook to offer me a range of options (all the lastgenre suggestions, but normalised against my hierarchy), or put in my own custom override. I think it would be cool if the plugin were to support such an option by default so people would have more control, but adjusting the code of the plugin is too difficult for me (as opposed to a simple hook shell script).

So anyway, that's some more input from my side. I will await the regex merge and if more info would be required from me, please feel free to let me know.

[arsaboo]

@spaceage64 This is great work and would be a vast improvement over our existing list.

I will leave the final evaluation to @JOJ0 but I hope we can get this (or some version of this) in the master.

[JOJ0]

In short, a first step would be: a total rework: "anything that is not an actual genre is a spelling correction" -> must move to aliases normalization (regex or simple match possible)

Only! genres, in one normalized spelling should be in genres-tree.txt !

@arsaboo or @spaceage64 if you want to start with that, any help appreciated!

[JOJ0]

Even for my own use case, I don't always want to use the genres suggested by the plugin, so I use a hook to offer me a range of options (all the lastgenre suggestions, but normalised against my hierarchy), or put in my own custom override. I think it would be cool if the plugin were to support such an option by default so people would have more control, but adjusting the code of the plugin is too difficult for me (as opposed to a simple hook shell script).

Please open a feature suggestion for that and explain in more detail how that would work from a user perspective. Maybe even with outputs of how your current hook scripts looks like when in use.

Thanks you!

[JOJ0]

In short, a first step would be: a total rework: "anything that is not an actual genre is a spelling correction" -> must move to aliases normalization (regex or simple match possible)

Only! genres, in one normalized spelling should be in genres-tree.txt !

@arsaboo or @spaceage64 if you want to start with that, any help appreciated!

Here I give you an example of the work that needs to be done:

This submitted section, needs the rework marked as comments:

- Ambient:
  - ambient # -> remove, is a case correction
  - ambient music # -> use normalize feature
  - Ambient Americana:
    - ambient americana # -> remove, is a case correction
    - ambient country # -> research, if this is a synonym, if yes ->  use normalization feature, if not can stay
  - Ambient Fusion: 
    - ambient fusion -> remove, is a case correction
    - Ambient Dub:
      - ambient dub #  -> remove, is a case correction
      - Dubtronica:
        - dubtronica #  -> remove, is a case correction
    - Ambient House:
      - ambient house #  -> remove, is a case correction

So after this work has been done, it would look like that (please keep 4-space-indentation in the file BTW)

- Ambient:
   - Ambient Americana
   - Ambient Fusion
   - Ambient Dub
   - Dubtronica
   - Ambient House

The additions to aliases.yaml would look like this (in a second step one could through this into an AI or regex101.com and find a smarter way to replace stuff. Note that my quick research proved that Ambient Americana and Ambient Country are actual synonyms, they are not there own separate genre but used interchangeably (correct me if I'm wrong please):

Ambient:
    - ambient music
Ambient Americana:
    - ambient country

Again, I have to think about how we solve the spelling issue, we will find a way, but for now, simply put the correct spelling into the genres-tree.txt -> Please open a new issue for solving this idea properly @spaceage64

Last step: Ensure genres.txt (the whitelist) contains the normalized form used in genres-tree.txt, in this example these need to be in there:

ambient
ambient americana
ambient fusion
ambient dub
dubtronica
ambient house

I know this is a lot of work and a lot of research and thinking but it is the only way. Sorry :-)

The guide I linked in my previous post might help to keep in mind when to use what feature:

Choosing the Right Tool
-----------------------

With multiple ways to filter and map genres, here is a quick guide on when to
use what:

- **Aliases**: Use these first to fix spelling variants and abbreviations (e.g.,
  ``dnb`` → ``drum and bass``).
- **Ignorelist**: Use this for error correction when Last.fm results are not
  accurate, or for precise per-artist or global exclusions (e.g., rejecting
  ``Metal`` for specific electronic artists).
- **Canonicalization**: Use this to automatically map specific sub-genres to
  broader categories (e.g., ``Grindcore`` → ``Metal``).
- **Whitelist**: Use this to finally limit your library to a predefined set of
  genres. When combined with canonicalization, the plugin will try to map a
  sub-genre to its closest whitelisted parent. Anything else is dropped.

[JOJ0]

Besides, even when only using lowercase in the code, my hierarchy is setup so that each Proper Case entry has directly under it a lower case variant used for matching. But as these are aliases, the Proper Case variant (i.e., the parent) should be returned instead. Perhaps I'm overlooking something, but it seems to me the logic already functions correctly. The only thing that would need changing, is getting rid of the additional Proper Case conversion the code currently includes. This would be the logic: lowercase any terms > regex lookup against aliases (all lowercase as well) > return Proper Case parent > done, no additional (potentially incorrect) conversion required.

We need to think on that matter! -> Separate issue please!

canonicalization is not spelling correction!