Improving feedback

At the core of this change:

1. Giving artists the ability to request "gentle" or "specific" feedback
2. Revamping the way curator feedback is rated
3. Improving our processes for finding curators who need a little bit of extra coaching

We hope these changes raise the bar for feedback on SubmitHub.

Problems we want to solve:

Feedback is a fundamental part of SubmitHub. It's also one of the most controversial.

Curators find it hard to write, and artists are often disappointed by what they receive.

Historically, we've asked curators to do two things when writing feedback:

1. Say something that proves you listened
2. Let them know why you're not gonna share it

About 70% of curators get this right. When they don't, though, we tend to hear about it.

Frequently-voiced concerns of submitters:

• "They said they loved it, but then declined for some stupid reason"
• "Don’t tell me what to fix. The song’s already done"
• "Is this even English?"
• "Wow, that was rude"
• "Why so vague? Did you even listen?"

Typical issues from the curator side:

• Too vague, eg "not a good fit" + "not the right sound for my playlist"
• Bad grammar / spelling
• Too critical when submitters don’t want critique
• Overly positive then declined, eg "ABSOLUTELY LOVE THIS" or "THIS IS AMAZING"
• Say the same thing to everyone
• Feedback is disagreeable or contradictory, eg "awesome vocals but not into the singing"
• Rude / overly honest feedback

New feedback options:

The most important step we're taking is to ask submitters whether they want "gentle" or "specific" feedback. By conveying these preferences to curators, we hope to cover most of the "too rude" or "too vague" complaints.

Gentle feedback:

• Recommended for artists who don't want critique
• Curators must still say something that proves they listened (eg, "The guitar solo was interesting")
• When providing a reason for declining the song, we ask that curators remind the submitter what they typically cover, rather than digging into the song itself (eg, "We typically focus on more chill music")

Specific feedback:

• For those with tough skin who can handle opinionated feedback from a stranger
• Submitters can make requests of the curators (eg, "What do you think of the chorus?")
• We let the curator know that the submitter wants them to be honest

No feedback:

• This option isn't actually new, but for those who don't want any feedback, they can choose to require that curators listen for a minimum of 90 seconds

Data so far:

As of this blog post, we've had 72+ hours of this new system. Artist requests appear to be breaking down as follows:

• Specific feedback: 63%
• Gentle feedback: 23%
• No feedback: 14%

Improving the way feedback is rated:

For the last ~5 years we've relied on a 5-star rating system to identify curators who are struggling to give feedback. This setup has some weird quirks to it, so we're getting rid of it.

Star ratings are too high:

The average site-wide rating is about 4.4/5. The vast majority of artists simply go through and rate all their feedback 5/5, which doesn't seem very genuine. Perhaps they're hoping that a positive rating will help their chances in the future? A parallel can be drawn to Hot or Not having an average rating of 69/100 - a very positive skew.

Low ratings scare curators from giving actual critique:

On the flip side, when a curator gets a 1/5 rating, it's more-often-than-not actually helpful / specific feedback getting shot down by someone who couldn't handle the rejection. And that effectively "teaches" curators that getting too specific in their feedback is a bad thing. So, they start sugar-coat everything in hopes of getting more 5/5 ratings, thus boosting their overall score. It's obviously not the desired effect, and pushes feedback in that "vague" direction many artists actually don't want.

Ideas for a new system:

To counter the above issues, we started collecting more "qualitative" ratings about a year back (eg, "rude", "vague", "technical", etc). At this point, we've actually got a pretty good database of information about each curator.

I haven't fully fleshed out the way the new system will work, but I think a good starting point will be to ask artists whether the feedback received met their expectations (eg, if you asked for specific, was it specific enough?). We can then steer submitters away from curators who might not match their preference.

Bottom line is that curators aren't gonna be seeing the 5-star rating system, effective immediately.

UPDATE MAY 27th - the new rating system is now live. More detail here.

Coaching:

For years now we've relied on two approaches to identify curators who struggle to give good feedback:

1. Leveraging artist ratings
2. Randomized samples

Going forward, we're going to revisit our existing mechanisms to see if there's room for optimization.

We've also put a good bit of effort into revamping the onboarding process for new curators. Historically everything's been done by chat, but going forward we'll be using a popup guide to pull off the task. This should give curators a better start when they join SubmitHub.

Surveying curators (tabled, for now):

This one's on the back burner because we're not sure it's actually going to work or add much value.

The idea is to conduct a survey of our curators to find out whether they've got technical expertise or not. There's obviously nothing wrong with not having any background in music creation (I fall into that camp). But I can see an advantage to knowing whether a specific curator can give you tips about audio engineering or production.

Again, this boils down to helping align artists with the right curators so that even if they are rejected, they walk away feeling like they've gotten something valuable from the feedback.

The reason we're gonna hold off on this one for now is because simply saying "this curator has audio engineering experience" doesn't mean their feedback is actually going to be good or helpful. And... do we need to verify those qualifications? Or can anyone claim they have them? So until everything's clear there, we're gonna hold off :)

Wrapping up:

Feedback is an important part of SubmitHub. If we can improve what artists get out of it, we can improve SubmitHub's overall value. Here's hoping these changes push things in a positive direction 💜