Quality Score Updates

The "Quality Score" is based on a curator's behavior on SubmitHub, and is not related to engagement.

December 2023 Update

The major change is that points are now considered on a linear scale rather than in steps. This should result in smoother day-to-day scores and less jumping around.

Here's a completely made-up example of how the previous system worked:

• Turnaround of 7 days = +1
• Turnaround of 5 days = +2
• Turnaround of 3 days = +3

Someone with 4.9 days got +2 points, but if they dropped to 5.1 days they'd only get +1 point. Under new system there's a linear sliding scale, so they'd get +1.95 points.

Metrics considered

In the name of keeping things transparent, we've decided to make the calculations of the quality score public 🔍

—————————

Methodology: Each curator starts at 50 points and earns or loses based on the below metrics. Their score is then compared to the max score. See more in March 2023 update below.

Note: There are a handful of additional metrics that are specific to a few niche cases and will not be shared publicly.

March 2023 Update

Historically, the quality scores have ranged from 5/10 to 10/10. As of today, they now go as low as 1/10. What this means is that someone historically at 8/10 has probably moved to 6/10, because the range has "widened". The underlying drivers of that score likely remain unchanged.

This change was brought about when I noticed that a curator with really negative signals (bad feedback, not sharing often) came back with a 8/10 score.

Old formula

Everyone starts at 50 points. Points are added or deducted based on the factors listed below. We then take the highest score, and everyone else's "rank" becomes their score divided by that higher one.

For example, if the highest score is 60, and yours is 55, your rank becomes 55/60 = 92% (or 9/10).

New formula

The new formula is similar to the old one, except that the divider has changed. Turns out the "minimum" score is typically around 30 points. Under the old scenario, you'd a score of 30/60 = 50% = 5/10. This doesn't seem like an indicative score for the literal lowest on the site.

The following part might get confusing, but works well in my brain 🤣

First, the new formula finds the minimum and maximum scores. It then looks at the difference, and uses this as the divider.

• divider = maximum - minimum
• rank = (points - minimum) / divider

Example

Let's assume our minimum is 30 and our maximum is 60. And that a curator has 55 points.

• old formula: 55 points / 60 maximum = 92% = 9/10
• new formula: (55 points - 30 minimum) / (60 maximum - 30 minimum) = 25 / 30 = 83% = 8/10

This new approach gives more variation to the potential quality scores, and gives curators more control over the ups and downs.

Primary factors considered

Some of the factors included in this measure include (but are not limited to):

• How quickly you share approved songs once they've been released
• This metric is referred to as "turnaround time" and excludes outliers, so one or two slow shares won't pull you down
• If you're a curator, anything faster than 5 days will earn you points, and anything slower than 10 days will start to hurt
• The bar is slightly higher for Spotify Playlisters, who have access to an "instant-share" feature that essentially means their turnaround time is often a matter of minutes
• How long you listen to your submissions
• This is based on premium feedback submissions (90 second submissions do not count)
• An average above 90 seconds gets positive points; an average below 60 seconds loses points
• Your response rate for submissions
• If you respond to 90% of your submissions within 48 hours, you've got nothing to worry about. Missing significantly more than that though can start to have a negative impact
• This also applies to standard submissions - if you have them enabled but don't respond, you'll lose points
• Your feedback ratings
• These are based on user reports and manual admin reviews (we manually review ~200 curators per week)
• You also get a small bump if you write really long feedback (we're talking 50+ words, of which only a handful do)
• The number of songs you approve
• Less than 1% proper shares starts to lose points; higher than 50% also starts to lose points (unless you have the "approves or refunds" option enabled)
• In a similar vein, having more than 20 ready-to-share songs also starts to bring you down a bit
• When you last approved a song
• If you've been actively responding to submissions and it's been more than a month since you actually approved anything, you might lose points (this does make exceptions for time off/breaks)
• Whether you share from your main "type" (eg, a YouTuber sharing primarily to YouTube is a good thing)
• The number of genres you have enabled
• Less than 15? Solid boost! More than 50? You're probably spreading yourself a bit thin :(
• Whether you're in the RGB program
• To get into this program you have to write quality blog posts
• If you accept standard submissions
• You don't get punished for not taking them, but you do get a small boost for actively responding to them, even if you've got a small daily limit of 5
• If you've linked to SubmitHub on your socials as your submission preference

In total, there are more than 25 factors taken into account when generating a "quality" score.

If you're a curator curious about how to improve, you'll see an option to view tips just below your score.

Site-wide distribution

Here's where curators end up in the mix.