tl;dr - The new model gets the answer right 1.6x more often than the previous model.
Genre detection on SubmitHub has become one of the site's most important features, not least because our automated Packages rely heavily upon it.
I've known for a while now that the model often defaulted to certain genres, including Dance pop, Old-school House, Ambient, and Trap. Any mess up means that we're not getting songs in front of the right curators, which has resulted in a lot of manual work by our team to make sure things are staying on point.
In light of that, I've made it one of my top priorities this year to roll out a better detector. And, after ~4 months of work, it's finally ready to share!
The detector has gone through a few iterations:
The MAEST model was pretty good - the best of its time - but it uses Discogs' genre names, not ours.
Last year I added a second, smaller model on top of MAEST that covered 85 of our own genres. That gave us 604 genres total, which I tried to line up against our ~260 genres.
But the added part had some major problems. It would often default to a "parent" or "catch-all" genre, rather than honing in on the specific sub-genre. On top of that, it was outright missing ~26 genres that we've either recently added or just never got around to.
I spent about three months trying to fix this by changing the model. This included:
I was banging my head against a wall. Bless my wife for putting up with me over those 2-3 months 😆
A hundred million Claude tokens later, and three things worked:
Step #3 was the breakthrough. I'd initially sourced the training data from Spotify's "The Sound Of" playlists, which come from the very excellent everynoise.com. I trusted that data too much. Turns out it was messy.
So, Dylan and I went through 265 genre playlists and ~45,000 songs to make sure it was all clean. A few curators also hopped in to help (thanks!). We're actually still doing this - so expect the model to improve over the next couple weeks.
Worth mentioning: another difficult step was building up a thorough arsenal of properly-tagged songs to test on. It was one thing to train the model to recognize indie folk - I needed to have ~50 clean indie folk songs the model had never seen to test whether it was actually work.
This is the good part 🎉
That's a 1.6x improvement, which I consider pretty massive given the old model actually wasn't that bad.
There are also 26 genres the old system couldn't recognize at all: things like alt-country, Motown, yacht rock, jersey club, dark pop, dub techno and international hip-hop. On 800+ "test" songs across those, the new model gets 57% right on the first guess and 76% in the top three, against zero before.
Best of all, the over-firing is largely gone. Under the old system, 20 genres each grabbed 100 or more songs that weren't theirs - Trap was the worst offender. Under the new one, not a single genre reaches 100. The worst is classic rock stealing 79 songs from other genres, which kinda makes senes when you consider slow rock, hard rock, and yacht rock and basically forms of classic rock.
I know some of the enterprising folks out there have made sure to select all the most-popular genres on this page. A lot of those genres show up on top because of the "over-firing" reasons I mentioned earlier in this blog post.
Going forward, I do anticipate that the mix of genres will change. For the better.
Curators who focus on niche genres should start seeing more accurate submissions, and perhaps more of them in general. Curators focused on generic "catch all" genres like "alt pop" or "indie pop" might see a dip in those.
Ultimately this is a good thing. But I do suspect it's gonna hurt for some of the folks who have come to rely on this big genres. This is just a hunch though - we won't know for sure until ~October.
Well, for starters, I'm now a genre expert. Every time I hear a song I turn to my wife and tell her the genre. At this point it's become a running joke. But it's no laughing matter: I've gotten damned good at it for a 40-year-old dad 😆
Either way, I'm not satisfied yet. Dylan and I still have a lot of audio to clean up. We've combed ~50% of the source playlists song-by-song, and going forward will focus on those that struggle (eg, post-grunge vs grunge).
If you feel like you got a terrible result or something was obviously wrong, hit that "Not useful" button and I'll take a look. Also... feel free to share your experience in the comments below. I bet we can fix it!