Recap: why do some organisations say their recent hires are worth so much?

Our 2018 survey found that for a second year, a significant fraction of organisations reported that they’d want to be compensated hundreds of thousands or sometimes millions of dollars for the loss of a recent hire for three years.

There was some debate last October about whether those figures could be accurate, why they were so high, and what they mean.conversation about the competitiveness of positions at EA organisations has also raised questions about the replaceability and overall value of these jobs although this post was drafted prior to that discussion. See also <a href=https://80000hours.org/2019/05/why-do-organisations-say-recent-hires-are-worth-so-much/"https://forum.effectivealtruism.org/posts/G2Pfpkcwv3bJNF8o9/ea-is-vetting-constrained">1, <a href=https://80000hours.org/2019/05/why-do-organisations-say-recent-hires-are-worth-so-much/"https://forum.effectivealtruism.org/posts/oNY76m8DDWFiLo7nH/what-to-do-with-people">2, <a href=https://80000hours.org/2019/05/why-do-organisations-say-recent-hires-are-worth-so-much/"https://forum.effectivealtruism.org/posts/EP6X362Q3ziibA99e/show-a-framework-for-shaping-your-talent-for-direct-work">3, <a href=https://80000hours.org/2019/05/why-do-organisations-say-recent-hires-are-worth-so-much/"https://forum.effectivealtruism.org/posts/HwBsTZEzGyAjQLoe7/identifying-talent-without-credentialing-in-ea">4.

" rel="footnote" class="footnote-link no-visited-styling" aria-label="Footnote">¹ In the current post, I outline some rough notes summarising the different explanations for why people in the survey estimated that the value of recent hires might be high, though I don’t seek firm conclusions about which considerations are playing the biggest role.

In short, we consider four explanations:

1. The estimates might be wrong.
2. There might be large differences in the value-add of different hires.
3. The organisations might be able to fundraise easily.
4. Retaining a recent hire allows the organisation to avoid running a hiring process.

Overall, we take the figures as evidence that leaders of the effective altruism community, when surveyed, think the value-add of recent hires at these organisations is very high — plausibly more valuable than donating six figures (or possible even more) per year to the same organisations. However, we do not think the precise numbers are a reliable answer to decision-relevant questions for job seekers, funders, or potential employers. We think it’s likely that mistakes are driving up these estimates. Even ignoring the high probability of mistakes, the implications of the data depend heavily on exactly what is driving the results. We are very uncertain about the magnitude of various considerations, so we recommend against leaning on these numbers when making career decisions.

Independently of this data, we believe that these jobs are sometimes very high-impact for some people. This suggests that finding out whether or not you’re a good fit can be valuable, even if most people won’t turn out to be. At the end, we sketch out some (weak) implications for job seekers. We hope to write about our overall views on the current job market in the effective altruism community in the future.

These are just rough notes and not a polished article, but I hope they’ll help to sum up the discussion and let the debate move forward.

A quick recap of the results

Here was the exact question used in the 2018 survey:

For a typical recent Senior/Junior hire, how much financial compensation would you need to receive today, to make you indifferent about that person having to stop working for you or anyone for the next 3 years?

Here are the raw results:

In theory, the ‘value-add’ of an additional employee to an organisation over three years (relative to what would have happened if the organisation did not attempt to hire) is given by:

1. The extra impact the organisation is able to have over those three years (compared to the counterfactual of not making a hire at all).
2. Minus the cost of their salary.
3. Minus the opportunity cost of management time (including the time it takes to run a hiring process).
4. Minus any other costs of hiring (e.g. changing culture).

If someone else would have been hired otherwise, then the marginal value of a particular hire can be approximated as the value of the person hired (as calculated above) minus the value that would have been created by the person who would have been hired otherwise.

Prelude: Are these figures even plausible?

The claimed value of recent hires seem surprisingly high at first glance, so you might consider dismissing them immediately. Before outlining some potential explanations for the figures, we’d like to suggest that the results are not totally implausible, prima facie.

In particular, the results may seem more reasonable when you consider the impact of some of the organisations in the survey. For instance, GiveWell has about 25 employees and moves around $100m per year towards its top recommended charities, which is $4m per employee per year. A typical salary is under $100k, so even if the opportunity cost of management time is $1m per year per person, the value created over three years by a recent hire who raised the average impact of their recommendations by a few percent could still be millions of dollars. (These are not my actual estimates of the impact of GiveWell employees, they’re just figures to illustrate that it’s possible for a reasonable calculation to come up with very high figures.)

Next are five considerations that could (partially or sometimes fully) explain why the figures are so high.

1. The estimates might be wrong

The organisations could easily be mistaken when they give their estimates. We think this is actually pretty likely.²

One way the respondents could be mistaken is by introspecting incorrectly. The question involves predicting how they would trade money against recent hires in a hypothetical. It could be that if a respondent were actually faced with the decision to lose a recent hire or gain such a large amount of money, they’d choose the money.

Another possibility is that the respondents are correctly predicting how they would behave given the tradeoff, but would be wrong to make such a tradeoff. In other words, the recent hires don’t add as much value as the respondents think they do.

It wouldn’t be surprising if the respondents were simply wrong. Our impression is that most of the answers were given with just a couple of minutes of reflection, and so mainly reflect a gut intuition. There’s not much reason to expect these intuitions to be accurate on average in this kind of domain.

If the respondents made one of these types of mistakes, what are some specific possible causes?

These estimates involve weighing up many hard to estimate factors, so it’s easy to let bias come in. For instance, one source of bias is self-promotion or overconfidence. The organisations might be systematically over-optimistic about the value of their hires, or want to give high figures since doing so inflates the perceived value of their hires or the respondents’ own work.

One particularly difficult aspect of the estimates is that you need to envision what would happen if you lost the hire. The loss is concrete, but it’s hard to envision all the ways you might adjust to it. This could lead people to overestimate how bad the loss would be. This might be analogous to how people typically overestimate how unhappy negative events will make them. In addition, it’s also hard to quickly envision all the useful things you might find to do with additional money.

Another difficulty that could lead to mistakes is accounting for the opportunity cost of management time. If you lost the hire, it would free up senior staff to do something else, which could also produce significant value. It’s not clear how accurately people were assessing the value of lost senior management time, and it would be very easy to forget to do so at all. This could lead people to overestimate the value of recent hires.

This said, there could also be biases and mistakes in the opposite direction.

For instance, it feels uncomfortable to give high figures and giving high figures could discourage donations (since it implies donations don’t achieve much compared to hires).

Another factor that could be easily overlooked is that additional hires not only have some short term impact, but they also speed up the organisation’s growth, and may develop into senior managers themselves, thereby allowing the project to make further hires more quickly years down the line. This growth component might be an important part of their impact, but it’s hard to estimate, so might get undervalued or forgotten. They may also leave and use their experience to do useful work at other organisations in the future. Again, this might be where much of the value of junior hires comes from, but may not be taken into account in the estimates.

Overall, the more suspect the estimates, the less you should update on the results and the more weight you should put on your prior. In our original discussion of these results, we said “unfortunately, we do not have very much confidence in the answers to these questions and would not recommend updating very much based on them.” Overall, we stand by that view.

We list other weaknesses in the survey here.

2. There might be large differences in the productivity of hires in a specific role

If the organisations believe that the people they have hired are significantly more productive in the role than those they could have hired otherwise, and the organisation has a lot of impact per staff member, then they will give high figures for their recent hires.

We expect large differences between recent hires and the next best candidate may often exist, for a couple of reasons.

Firstly, it’s very hard to get a precise estimate but we believe that the (ex-post) output of different workers in a field usually varies a great deal, especially in skilled jobs. Some people who have studied productivity have argued, for instance, that researchers vary 100-fold in their output.

The differences also get more extreme when you draw from the tail of a distribution.³ An organisation that only hires a couple of highly skilled people each year will be able to choose all its hires from the tail of its hiring pool, so differences among their best candidates will typically be larger.

If some fraction of these differences in output are predictable by an organisation, it could reasonably think that good recent hires are several times more productive than the next best person in their hiring pool.

The situation in effective altruism organisations might be even more severe than with other skilled jobs, if they require an especially rare skill set. Most roles require knowledge of many different aspects of effective altruism, and the pool of people who have that is very small. If the pool is already depleted, then there could be large differences between the best candidate and the next best candidate.

What’s more, the differences in value-add get even larger still when you also consider costs as well as output.

The costs of hiring are to some extent fixed. For instance, one of the main costs of hiring is that you use up the time of a manager. Let’s have a simple model where each hire takes up 10% of the time of a manager, and we value this at $100,000 per year. (These are just illustrative figures and should not be taken as estimates.)

Then consider three potential hires:

A. Produces $150k per year.
B. Produces $110k per year
C. Produces $100k per year.

When we subtract the $100k of management time, the value-add is:

A. $50k
B. $10k
C. $0

What’s surprising about this is that there was only a 36% difference in output between A and B, but their value-add varies by a factor of five. The 50% difference between A and C means that C has zero value-add.

In fact, the situation is even more extreme again, because different hires also vary dramatically in their costs. A great hire who’s trusted and independent might take up almost no management time, while someone who ends up being a bad fit could take up a huge amount of management time.

Likewise, a great hire can improve the culture and make other staff more motivated, while a bad hire can easily de-motivate everyone else.

While we’re very uncertain of the magnitude of the factors in this section, we believe they could plausibly contribute to very large differences between the value-add of recent hires and the next best candidate, especially among senior roles.

3. The organisations may be able to raise funds easily

If the organisations surveyed are able to easily raise funds, they’ll be willing to give up large potential donations to retain recent hires, and will give very high answers to the survey question. We only asked the organisations to quantify the value-add of additional donations to their own organisation. If some organisations are already able to consistently raise funds whenever they want to expand their budgets, then they may be nearly indifferent to additional funding — they simply can’t do much else with more money and if they do come up with another effective use of money, they can be confident they’ll find it from another source.

In theory, the respondents should have considered the possibility of donating un-needed funds somewhere else (at least to the extent that they are trying to maximize amount of good done overall and not biased toward their own organisations).⁴ But it’s not clear that, even if their organisation would regrant the money in this situation, the respondent would consider this possibility when responding to a question framed around the tradeoff between retaining a recent hire and donations directed to them in particular.

This would mean that the figures would mostly reflect organisations placing a low value on marginal donations, as opposed to a high value on staff. The figures would then be very poor evidence for the questions we actually care about – like how much value potential donors and staff should place on direct work.

It does suggest that earning to give and donating to that particular organisation is less useful, but in that case people can direct their donations elsewhere instead.

In the next version of the survey, we could try to ask what a donor should be willing to pay, rather than what the organisation would need in compensation to retain a hire. We could also try to ask about the value of hires in a different unit.

4. Retaining a recent hire also allows you to avoid the cost of running a hiring process

The survey question asks about the value of retaining a recent hire, which is different from the expected value of deciding to hire a similar candidate. Finding a new person requires running a hiring process, which can be expensive. It takes a long time and a lot of effort to find the right people, train them, measure performance and build trust. We list some of the costs of hiring here, and also see this post by GiveWell.

In a recent post reflecting on Open Phil’s 2018 generalist research analyst recruiting, Luke Muehlhauser wrote that “despite our time-intensive application process and trial period, in most cases we didn’t feel we had a good read on which candidates would be a good fit for our generalist RA roles until roughly 2 months into the in-person trial.”

Let’s say a hiring process took up 1 month of senior staff time. If an organisation values this at $100k, then anyone who is already through the hiring process is worth $100k more to the organisation, when compared to people who haven’t been through yet.

The 1 month of senior staff time and $100k figure are not my estimate of the actual cost of hiring. It’s just an illustrative figure to show that this could drive up the value-add figures of recent hires. In practice, the cost of hiring varies a great deal between organisations, candidates and situations.

A potential staff member who has already been identified, is trusted and can hit the ground running has fewer costs, so is worth more to the organisation.

Hiring requires several sequential steps, which means that if you get unlucky, there can be years between identifying a need on staff and successfully onboarding someone who is a good fit. Search, selection, and training can take over a year. If you then discover the person wasn’t a good fit after all, you have to start over. This serial dependency can create bottlenecks to hiring even for an organisation that is committed to expanding. If the organisation is growing rapidly, it can end up hiring far fewer people than it would like to. This increases the value of people who have already been hired.

If the cost of hiring accounts for a lot of the value of recent hires, then this increases the value of retention at impactful organisations (and is a consideration in favor of staying at your current job if you’re already at an impactful organisation where you’re a good fit).

This could also partially explain why some organisations placing a high value on recent hires don’t continue hiring. When considering the value of adding another staff member, they have to subtract out this cost.

If the cost of hiring is a major reason for high figures, then we could say that vetting potential hires is a key bottleneck for EA.

Do the survey results have any decision-relevant implications?

Which of the above considerations play the biggest role in driving the results? I’m unsure. I think each one has an effect but don’t know their relative magnitudes. This is one reason that decision-makers — including job seekers, staff at impactful organisations, potential employers, and funders — should not rely on the precise magnitude of these figures to inform their decision-making. Their implications depend heavily on the (unknown) size of the effects we’ve discussed.

The group these figures do have some relevance for are the most recent hires of these organisations. Inasmuch as the willingness to sacrifice donations to retain them is high, that suggests they shouldn’t quit if their alternative would be earning to give to the place they work. However, the large figures might be caused by the organisations surveyed having few funding constraints, so they can’t tell us much about the relative benefits of working for one of these organisations or earning to give and donating elsewhere.

The group for whom these figures are second most relevant are those choosing between working for a particular organisation, and earning to give to that same organisation. Inasmuch as these figures are very high that suggests that they should expand the options they are choosing between to include earning to give and donating to other more cash-strapped projects.

Beyond that, my personal view is that there’s a good chance the figures are wrong, so they shouldn’t be a major part of anyone’s process for making career decisions. Instead, you could lean on more robust considerations, such as whether the organisation is high-impact, your degree of personal fit with the role, how much career capital you’ll gain, and so on.

I also think that the high cost of hiring processes and ease of fundraising for some organisations increase the figures, making them less useful for most decision-relevant questions.

Nonetheless, for other reasons, I do think (i) some of the organisations in the survey have a lot of impact and (ii) there are large differences in productivity between hires, such that some staff have a very high value-add. This means that at least some positions at these organisations are likely to have a very large impact.⁵

This would suggest that, if these jobs are among your shortlist, it’s valuable to find out if you might be able to end up as one of these high value-add staff.

However, it’s important to do this while bearing in mind that the highest impact positions are not all in these organisations. Working at an effective altruism organisation is just one among ten priority paths that we recommend especially highly. There are many high-impact positions in other types of organisation, so it might be even better to focus on testing your fit elsewhere. The rest of this section is written for readers who think working at effective altruism organisations is in their shortlist of long-term paths.

It’s also important to keep in mind that the base rate for any application being accepted is under 10% – sometimes well under. In addition, there are only a handful of these organisations, so only something like 20 to 40 positions open up in a given year. This means that even someone who has promising fit can’t be confident of landing a position. In the same way that no-one looking for their first consulting job would plan their career around working at one specific firm, no-one should plan their career around getting a job at these 20 or so organisations, which between them have fewer job openings than a large consulting firm.

However, that doesn’t mean it’s not worth applying for these jobs. Since there could be a high upside if it works out, it can be worth making applications or learning more about the area, so long as you can do so in a way that doesn’t set back your career in other areas.

For instance, we sometimes come across people who almost didn’t apply to these positions since they thought they had no hope of landing a job, but then turned out to be successful. It would be good if more people like this applied; however, we’d never encourage someone to only apply to jobs at effective altruism organisations. See making applications as an attempt to learn about personal fit, and apply to other types of jobs as well.

Similarly, if you’re not able to get a job right away, it could be worth taking a position that builds career capital for these positions, but we’d usually only recommend doing this if the position also builds your career capital for other paths. For instance, doing think tank research both prepares you for research jobs at effective altruism organisations and opens up top paths in policy.

One final point to remember is that the high dollar figures don’t necessarily imply that it’s easy to get a job at these organisations. Rather, the high figures might reflect the fact that very few people are a good fit for these jobs, making it harder to get jobs at these organisations.

In brief, exactly how to respond to this situation will vary a lot from person to person and it’s hard to give generic advice. If you’re unsure about the next steps for your career, you can see a summary of our general advice for comparing your options in our decision process.

Appendix I – A possible alternative survey question

At some point we will have the opportunity to survey organisations about the financial value of their hires again.

Ideally the answers would be informative for someone deciding between i) earning to give and doing direct work at the organisations surveyed; ii) someone choosing between working at different organisations surveyed; iii) someone deciding which of the organisations surveyed to give to; and iv) someone deciding between direct work at those organisations, and other direct work options elsewhere. Unfortunately, like most social science, this is not straightforward, and so we will probably have to focus on one of these groups.

Below is a very rough draft of one version of the question we are considering asking. We hope it would be more decision-relevant than the question used above, but we haven’t yet had time to pilot it or vet it for any issues:

Imagine that sometime in the next year you are about to hire your next junior (senior) hire. A genie appears and offers you the following choice. You can have one of the following:

1. The genie will create a person and applicant for the job from thin air. They will be as much more productive (in % terms) than the next best applicant in the pool, as your last junior (senior) hire appeared to be at the point when you were evaluating whether to hire them. This person will live out the rest of their life like any other staff member, and may well go on to do other useful work outside of your organisation later on. You should consider the benefits of that for the world as well.

2. The genie will distribute $X among whichever organisations or people you nominate – which can include you and your organisation – to be used to improve the world as much as possible. Consider all the benefits for the world this would generate.

At what value of X would you be indifferent between these two options?

We would value feedback on the ways this could be a good or bad question.

Some issues to keep in mind include:

• A hire is less useful before you’ve tested them (ex ante) than after you have and decided to keep them (ex post).
• Respondents are better able to respond to concrete questions about the past and specific people than hypothetical future ones.
• We should consistently allow people and funding to flow out to their next best opportunities in the rest of the world if an organisation finds it hard to scale up.
• We should compare adding staff and adding money, or removing money and removing staff, to avoid getting confounded by people valuing losses and gains differently. This is another way the question we used previously was not ideal.
• The more convoluted the question becomes, to deal with the problems described in this post, the less intuitive it is to answer, which can itself make the results less reliable.