How good are the best?
In complex jobs, the best 10% of workers produce 8 times as much as the bottom 10%.
Imagine you’re offered the top job at SCI one of the world’s most cost-effective charities should you take it?
SCI’s historical annual budget has been just under $10m (1). With $10m, SCI is able to provide 20 million children with medicine that prevents neglected tropical diseases each year (2). We’d expect this to lead to 250,000 years of extra healthy life per year (3), as well as to yield long-term education and economic benefits to the millions of people in the affected area (4).
A 10% improvement to SCI’s efficiency means treating an extra 2 million children and creating an extra 25,000 years of healthy life per year. That’s roughly equivalent to saving 800 lives per year (5). So, a small improvement to SCI’s efficiency is a big deal. The same would go for someone who could fundraise an extra 10% for SCI.
Whether you could increase SCI’s impact by 10% is a tough question, and given the organisation’s extreme efficiency, we have reason to be sceptical. But one issue that bears on the question is the following: how much do different employees typically differ in output?
There have been many studies looking at this very question, across a wide range of jobs, which are summarised in a meta-study by Hunter, Schmidt and Judiesch (6). Output is measured in a variety of ways. For salespeople, it’s the dollar value of what they sell. For doctors, it could be the number of patients seen and treated. Other studies have been done with standardised tests, supervisor ratings and many other metrics (7). I should flag that it’s not clear how well these metrics correlate with the real value produced by jobs, but I’ll run with them for now.
What they found is that in low complexity jobs, workers’ outputs do not vary much, and the best worker is usually not much better than the average worker. As the jobs become more complex however, there’s more and more variation, and the difference between the best worker and the average grows. For example, in low-complexity jobs the top 10% of workers produce 25% more than the average, and 75% more than the bottom 10%. For high-complexity jobs, such as professional and sales jobs, the difference is much larger. The top 10% of workers produce 80% more than the average, and 700% more than the bottom 10% (8).
Taking these findings literally suggests that the bottom 3% of workers in professional jobs have negative output. Hunter, Schmidt and Judiesch believe this is unlikely, and interpret it to mean that the distribution is not a perfect bell-curve, but instead stops around zero output (it’s probably log-normal instead). I agree that it’s unlikely that someone’s direct output could be negative for long before they were fired. Imagine if a doctor killed more patients than he treated. But if we think about all the indirect effects someone can have on an organisation, like decreasing team morale and consuming lots of supervisor time, then it’s not implausible that some people have an overall negative contribution.
The methods used to measure output often don’t take these indirect effects into account. They tend to focus on what each employee did directly, e.g. the number of insurance contracts sold or the number of patients treated. This would suggest the studies significantly understate the differences in output that really matter.
The crudeness of the metrics will be another reason why the figures might tend to underestimate the true variability in output. For instance, ‘the number of patients treated’ gives some indication of the output of different doctors, but some treatments are better than others. Curing 100 patients of sore throats is not as good as curing 100 patients of cancer. So, there another important type of variation that’s being missed.
So, among charity managers and fundraisers, it would be normal for a good one to be about twice as good as an average one, and many, many times better than a bad one. Being 10% better than your replacement at the top of a charity, therefore, could be quite achievable. And very high impact.
You might also be interested in:
References
(1) The total budget from 2002-2010 was $68m.
http://givewell.org/international/top-charities/schistosomiasis-control-initiative##Overallspendingbreakdown
(2) The annual cost of treatment is about 50p/person http://www3.imperial.ac.uk/schisto/whatwedo
(3) Using $40/DALY http://blog.givewell.org/2011/09/29/errors-in-dcp2-cost-effectiveness-estimate-for-deworming/
(4) http://givewell.org/international/technical/programs/deworming##Developmental
(5) Using the standard conversion: preventing 30 DALY = saving one life
(6) Hunter, J. E., Schmidt, F. L., Judiesch, M. K., (1990) “Individual Differences in Output Variability as a Function of Job Complexity”, Journal of Applied Psychology
http://psycnet.apa.org/index.cfm?fa=fulltext.journal&jcode=apl&vol=75&issue=1&page=28&format=HTML
(7) Note that the study does correct for the fact that we can’t perfectly measure output, because we can only take a limited sample of what the worker does
(8) More accurately, output follows a normal distribution (though sometimes with positive skew). For low-complexity jobs, the standard deviation of the distribution is 15%. For high-complexity jobs it raises to 45%. The most variation is found in life insurance sales, where output has a standard deviation of 110%!