When Is A Star, Not A Star?

I read an interesting post from Nate Silver this week. In it he questioned the need for Uber’s 5 star rating system when a simple thumbs up or thumbs down could suffice. His reasoning was that Uber’s drivers and passengers over time have re-defined the way a scale measures the quality of a journey.

The premise of a scale is to rate something from good to bad, poor to excellent, agree to disagree; essentially whatever you wish but anchored at both ends by opposing factors. The world isn’t black and white, so we need a grey in the middle which provides a more detailed view of sentiment beyond a 2 point yes / no answer.

For those unaware, Uber has a 5 star rating system to rate drivers after a trip. Seems simple enough, he/she got me there in the quickest way, but their car wasn’t clean….4 stars, they offered me a bottle of water and a phone charger but got lost….3 stars, everything was great…5 stars. But you will soon learn from a driver that it’s even simpler (according to them). I’ve lost count of the number of trips I’ve taken where the driver has told me that I’ve received 5 stars from him (passengers are rated on a 5 star rating as well by drivers), and am requested to reciprocate with the same rating.

Everyone is different and we all have differing opinions of what dictates good service. There are those who never give a perfect score, ’because nobody’s perfect’ or others who might be unnecessarily harsh because they were offered a bottle of home brand water and not their preferred Evian.

Be seeking out a ‘5 star journey’ every time, drivers over time have created a benchmark level where 5 is the required standard and only should anything go wrong, would a passenger ever think of rating their driver any lower. Essentially, any lower than 5 is a fail. Through conditioning passengers by trying to influence their ratings, they have inadvertently given rise to a 2 point scale to be judged by, pass or fail.

While Uber has realised this, they have contributed further to the issue. For any rating that is not a 5, passengers are asked to identify why their journey was not up to standard from a list of potential factors. If I rate my driver as a 4 (which most would still think is a good score), I’m asked what was wrong with the trip. By doing this, I’m, conditioned further to believe that a rating under 5 is a bad thing. It does appear to give weight to Nate Silver’s ponderings, why bother with a performance scale when anything other than ‘perfect’ is seen as failure and when a score of 1 is treated in the same manner as a 4.

The learning to take from all this, is the impact that bias and conditioning has on capturing what should be impartial data. For drivers, through seeking to influence the passenger by setting scoring parameters to up weight their own performance and from Uber themselves by setting expectations that anything below a 5 start rating is a fail, this has created an artificial and dichotomous viewpoint on something which may not always be a case of black and white.

What this does is raise the point for your own performance management systems. Customer satisfaction or NPS (Net Promoter Score) are a well utilised means of capturing sentiment and are often factored into performance KPI’s. So think about the following:

  • What influence do staff have over the process? Can they bias customers while still in contact with them? While intentions may be good, it may artificially influence what should be impartial scores.
  • How frequently are you requesting feedback? By understanding the system, consumers can be influenced by lengthy follow up questions for a particular score given. Will they circumvent this by giving an untrue representation of their experience in order to speed through the process? For example, I know that I’ll be asked to give feedback if my NPS score is a 6 or below, so I’ll give a 7 and not have to worry.

I would ask you to rate this blog but I’m not sure what measurement I’m looking for.


By Chris Binney

Leave a Reply