How to add value in international development: an interview with Eva Vivalt

By · Published June 19th, 2013

I recently interviewed Eva Vivalt, who works for the World Bank and is the founder of AidGrade, a new organisation that evaluates and recommends different development programs on the basis of effectiveness. AidGrade’s mission is “to improve the effectiveness of development efforts by understanding and encouraging what works using rigorous, actionable and engaging evidence.” You can find out more about AidGrade on their website.

We talked a little about her career background and what AidGrade is doing, as well as what career advice she might have for people with similar aims in poverty reduction and international development.

Tell us a bit about your background and how you got to your current position.

I was interested in the idea of “helping as much as I could” from a very early stage. I did my undergraduate degree in philosophy, but realised partway through that this probably wasn’t the best way for me to figure out how to help the world. Not that philosophy isn’t a great way to do this for some people, but personally I thought I could better get this through studying international development. So I did a Masters program in Development Studies at Oxford, which was fantastic. The course is a real mix of different disciplines, which ensures you don’t think too narrowly early on.

I realised I particularly liked the way economists were thinking about issues and the way they arrived at conclusions. I felt like I thought in a similar way, and so this seemed like a good direction for me to go in. So I went to Berkeley to do a PhD in Economics. My goal coming into the program was to get out quickly and go and work for the World Bank.

Once there, I kept looking for good ideas for something that I could do. Eventually AidGrade seemed more and more like it was that thing. I found a great group of people in DC who really helped: we all gelled around this idea and built it up from there.

What would you say were the key decision points in your career? How did you deal with them?

Looking back, there were three key decision points: The first, choosing between philosophy, physics and development. The second, whilst I was finishing off my PhD at Berkeley, was deciding whether to go and work for the World Bank. And the third was choosing to start AidGrade.

The first came up when I realised I wanted to move away from philosophy. I wavered between changing to physics rather than development, mostly just because I really like physics, and I do think it’s an important field. But I realised that I was better suited to development, so could personally make more of a difference there. This made the decision fairly straightforward. I would probably be average in physics, so it would be harder to contribute much. In development it also seemed like there was more scope for doing things: there’s a lot of money and interest in the field and the moment, and you can move quite a lot of resources.

The second key decision was whether to go to academia after my PhD or to go and work for the World Bank. To be honest, there wasn’t much chance I wouldn’t have accepted the World Bank offer, and I didn’t examine this decision much. Although the outcome was good, I could have done my homework a lot better; I didn’t know much about the World Bank before joining it. I’d spent one summer in its research division, but the World Bank is divided into research and operations, which are two very different worlds. This wasn’t something I understood at the time.

Deciding to start AidGrade was the third key decision and definitely the most difficult one, because in a sense it was a point of no return. It does a lot of meta-analyses, and if you do meta-analysis, people look at you a bit funny. There is not much prestige in it, as researchers prefer to write the first paper on a topic to get a better publication. You are also suddenly in the thick of great wars between different economists. Further, people doubt your intentions because it is a bit “pop.” In general, there are huge risks in pursuing a venture when it’s hard to get a sense of how well it will work, whether it will be successful or not. I didn’t have much to compare it to, and so not much with which to form beliefs! All that said, I’d gotten to a stage where I wanted to narrow down my options and pick one thing to really push forward, and AidGrade really seemed to be the best option. It’s also not a venture that rules out other things: I can still continue my academic research, for example. I did try to test the water as best I could by getting as much feedback as possible from other people. Talking with people tends to be much more helpful than doing research on your own, although different people always have different incentives, so you have to be wary of this.

Tell us a bit more about AidGrade. What exactly are you doing, and why?

We realised there was this big gap between people who have really studied development economics in depth, and people doing analyses of aid programs. Most of the people who can do this kind of analysis don’t want to do it, and those who want to do it, sadly aren’t always very good at it. Another thing we noticed is that you can break down the components of a meta-analysis into small chunks, get a lot of people involved, and then do it much more cost-effectively. A lot of people do do meta-analyses – the Cochrane Collaboration are a good example – but the way it’s been done in the past can be problematic and isn’t as efficient as it could be. One group of researchers spends a lot of time gathering papers, screening the papers, and then comes up with an estimate based on this set of papers. But if another set of researchers want to question these results – perhaps because they think they should have used a different set of papers – they have to start the whole practice from scratch.

So what we want to do is get the “superset” of papers people could be interested in when assessing different aid interventions and build a database that anybody could use afterwards. This means you wouldn’t have to keep reinventing the wheel every time you want to do a meta-analysis on a subject: you could just go and use this database. We’re also aiming to code up different variables that people might want to filter on: so if you’re interested in, say, school meals programs in a particular location, for a particular age group of children, you can narrow down the studies to these. You’d also be able to filter on whether the study used particular methods, reported various things, or scores well on various quality characteristics.

Could you explain a bit more how AidGrade differs from other existing charity evaluators like GiveWell?

To start with, GiveWell itself doesn’t actually do meta-analyses; they read up on meta-analyses done by others. I think that’s great, but in some subjects, there aren’t existing or up-to-date meta-analyses. And there are flaws in the general approach to looking at past literature. Say I take 10 studies, 7 of which find no effect, 3 find some positive effect, and I conclude there’s not much evidence of any positive effect. Statistically speaking, this isn’t a good way of going about it because it might simply be that those 7 studies didn’t have the right sample size, and if you combined them in a meta-analysis in the right way you’d see a stronger effect.

It’s also difficult to analyse in what context different programs do and do not work: the “average location” doesn’t really exist, so we can’t necessarily say a program has a given effect “in general.” The idea is that our database will allow you to drill down into a bit more detail, do a meta-analysis on any subset of papers, which I think is really important.

I don’t mean to put GiveWell down – I think there are a lot of things they do really well. I just think there is definitely value added by what AidGrade is doing. We’re not a competitive venture in any way: both organisations benefit from each other’s presence in this field.

Why is it you’re looking at a very broad range of organisations and interventions, as compared to someone like GiveWell who only recommend a very few?

We don’t really feel comfortable with the idea of recommending a very few organisations or interventions. This is because when you have a great deal of uncertainty about something, you actually do better if you select more options to mitigate your risk. The same reasoning explains why people diversify their own investments: and if you diversify for yourself, you should probably diversify for other people, right? On an individual basis, it maybe doesn’t matter so much: if you have a number of individuals pursuing different things, it probably does balance out in the end across individuals. But whilst we do have some sense of some programs being better than others, and we do want people to be donating to the most effective causes, I feel a great deal of uncertainty remains: enough to make it worth diversifying a bit more.

That makes sense. But given that we do at least have some evidence of certain causes or programs being more effective than others, isn’t there a risk of diversifying too much?

I would definitely agree that we should support some programs over others. I wrote up a fuller discussion of the issue here. In particular, I would like to emphasize that given that people do contribute to a wide variety of causes, in effect the status quo is already very diverse and donations could use more direction. I’m happy to talk more about this in a further post or in comments if people have questions.

How did you decide which topics to focus on initially?

First, it was important to select topics on which there would actually be an adequate number of studies, so we did some initial searches to whittle down a long list. We’re actually doing an impact evaluation of our own work, so that has also affected how we have selected topics. For the impact evaluation, the most relevant factors that we thought could affect the results of the study were: the number of impact evaluations on each topic; the initial level of public interest in each topic; and the initial level of institutional funding for each topic. We held a public vote and took the winner as a topic for investigation, and the other topics that survived the preliminary searches were matched to each other and then randomized within the pair. The AidGrade Process Description contains more information about the procedure.

As a general rule, if one is solely seeking to maximize value of each meta-analysis, one should target those areas where there are a lot of studies but limited synthesis of the data, as well as areas which are of high policy importance and which could be changed by new information. (I am actually preparing a document for someone at the World Bank at the moment summarizing the areas where we know the least relative to spending.) We plan to use this kind of targeting more going forward.

So what are AidGrade’s key aims, in the short term and the more long term?

We’re still compiling this database, so that’s a key aim for the short term. We want to eventually have all impact evaluations in international development: there are only several hundred out there really, so I think this is quite plausible as a short term goal, and we should be covering it quite soon.

Beyond that, there are some interesting research questions that can be answered with the same database. Unfortunately, there are large issues of credibility within research, and that’s a field that is really ripe for new work. There are now all these different ventures in this field, but there’s still space here I feel. People are mining their data inappropriately: for example, running millions of regressions, finding something significant in one, and just reporting that one. That’s not how hypothesis testing is supposed to work, so there’s really a problem with that. Then there are all these studies where people have tried to replicate past results and failed. I heard a story about a study a while back where they tried to replicate 45 articles in a journal, and 43 of them couldn’t be replicated. It’s ridiculous. So we’re thinking about how we could potentially move in this direction, in a way that’s constructive and helpful and other people aren’t already doing. There are a few ideas there, but it’s all very tentative at this point, and that’s more of a long-term thing. We’re still largely focusing on getting these impact evaluations in our database.

What do you think are the biggest limiting factors in fighting global poverty at the moment? Lack of good researchers, lack of funding, lack of good people in policy, or something else?

It’s not necessarily that there aren’t good researchers, but I guess I feel like the talented researchers don’t necessarily do the research that is important. It’s more of a mismatch in that way. A lot of people who are interested and really want to make a difference don’t actually seem to end up pursuing this. There is a definite need for people who say okay, I’m interested in these subjects, I’m going to undergo lengthy training to understand the issues, write and read academic papers, but then focus on the topics that are most pressing rather than those that are “hot.” I don’t mean to disparage academia, because there are a lot of really fantastic people there, but at the same time, there are all these competing pressures. The talented people need to find ways to avoid those pressures.

There’s also the fraud issue unfortunately: hopefully this can improve with better incentives. And even if researchers are focusing on the important topics, there’s still the issue that you have to be able to convince policy makers, and they don’t have the right incentives either. They can always find someone who calls him or herself an “economist” and will support their views; it’s a very unfortunate situation. There are numerous steps along the way here where these things can go wrong.

A bit closer to home, what are the biggest limitations AidGrade is facing as an organisation?

One of the biggest limitations we face is that people don’t naturally care about the fine points of rigorous analysis. It’s not sexy. And we don’t have a big PR movement that makes flamboyant statements or strokes people’s egos to attract media attention. If nobody cares about your results, why bother? That’s a pretty big limitation, though one we hope will be mitigated in the future by the slow but steady turn to data as well as great partnerships and growing publicity. Large institutions also generally see the value in data and in meta-analysis in particular.

Another answer is money. We don’t earn profits, so we need to rely on constant outside funding for our work. Sorry this is such a lame answer to this question, but it’s true – it’s a constraint and we could be doing a lot more. Overall, though, I think we’re doing quite well. As we say on our website, AidGrade isn’t intended to be a cure-all. However, it has a very valuable role to play and one that isn’t currently addressed by any other organization.

How can people get involved in AidGrade? What are you looking for, what skills and background would you need?

It’s all quite open at the moment, there’s a real variety of things we need doing. Because we try to break down the meta-analysis process into individual components, that leads to a wide range of tasks, there’s really something for everybody. If you don’t have a lot of background, you can help screen titles for papers. If you have a bit more sophistication and time and you want to get a bit more in-depth, you can read through some of the academic papers in more detail and help code up the characteristics of the paper. Going further than that, you can help analyse the data. Plus the typical variety of roles you’d find in any organisation: social media, web development, fundraising. There are really many, many roles.

It’s easy for people to get in touch through the website, where there’s a lot more information on all of this than I can describe. If you’re interested, there are some email addresses on the site and some forms: any of those channels will do!

What do you think are other top jobs in this area?

It’s a tough question: I think there should be more innovative work. I’m a big fan of all of these innovation competitions and these kinds of things, I think they’re fantastic. I think it’s unlikely the biggest opportunities are in particularly large organisations that have a lot of bureaucracy, unfortunately. When you have a very large organisation which is somewhat ineffective, it’s very entrenched in the culture, but then again it also depends on the individual.

At the same time, I don’t want to encourage a “too many cooks”-style problem: people always want to go off and start their own idea, which doesn’t always work. But there may still be a lot of low-hanging fruit at the moment. I’ve spent some time in the San Francisco Bay Area and I was quite impressed by some of the ideas coming out of there.

More specifically, I wish there were more advocacy for freer migration. I think that’s an area which is rather under-served compared to the evidence on it.

To wrap up: what would your biggest pieces of advice be for someone in the early stages of their career passionate about fighting global poverty? What do you wish you’d known?

At some points, pursuing something just trying to helping others can be a long, hard slog. A lot of people will say what you’re doing is admirable, but at the same time, you’re kind of on your own on this path. There’s not much support, and what you’re trying to do can be quite risky. So I’d just recommend firstly being cautious, but also keeping in mind that it’s OK to fail, and making sure you have a good support system in place.

Aside from that, I would advise trying to learn as much as possible early on about the different areas you could possibly pursue. Originally, I went into the economics PhD because I knew I wanted to be an activist, but I wanted to figure out what I wanted to be an activist for! I think that gaining a lot of knowledge early on is a really good idea. Learning is an ongoing process, but when you’re relatively young is the best time to invest as much as possible in figuring out what to do with the rest of your time. Don’t just jump into something – you’ve got the whole of your life.

You may also enjoy: