Transcript

Robert Wiblin: Hi listeners, this is the 80,000 Hours Podcast, where each week we have an unusually in-depth conversation about one of the world’s most pressing problems and how you can use your career to solve it. I’m Rob Wiblin, Director of Research at 80,000 Hours.

Today’s episode is going to be a cracker for those of you who are interested in using evidence, social programs, economics, or ending poverty. I talk to Dr Rachel Glennerster who is currently the Chief Economist at the UK’s Department for International Development – otherwise known as DFID. Before that she was the Executive Director of the Abdul Latif Jameel Poverty Action Lab – otherwise known as J-PAL – from around its founding in 2004 until the start of 2018.

I was due to interview Dr Glennerster on stage at Effective Altruism Global London, but sadly spoke to so many people at the conference that I lost my voice. Fortunately, Nathan Labenz was a real hero and at the last minute took my notes up on stage to fill in for me, so the first 28 minutes are Nathan’s interview with Dr Glennerster at that event.

Fortunately, I was able to find an hour with Rachel once my voice had come back, to ask some questions Nathan didn’t get to, follow up on some surprising things she had said, and find out more about what approaches to ending poverty she thinks are most underrated.

Many of you will have listened to episode 30 with Prof Eva Vivalt – if so, you can especially look forward to Rachel offering a quite different interpretation of the results Eva outlined in that episode.

Just before that though, I want to draw attention to two other podcasts you might be interested in subscribing to.

Vox.com recently started an effective altruism focussed podcast called Future Perfect. It goes along with a whole new effective altruism focussed section of their site, also called Future Perfect. Blogger Kelsey Piper recently joined that team and she’s been writing some great stuff, most recently about why socially responsible investment is probably overrated.

The host of the podcast is my friend and journalist Dylan Matthews. The episodes are about 20 minutes, and cover a wide range of topics – in the first season they looked at opportunities to improve the world through humane fish slaughter, open borders, geoengineering, voluntary organ donation, and putting lithium in everyone’s tap water, among others. You can subscribe by searching for Future Perfect in your podcasting app, or going to vox.com/futureperfect.

The second is the Future of Life Podcast which focuses primarily on catastrophic threats to the future of humanity as a whole.

Lately it has been getting more similar to this show, featuring long-form interviews with experts on issues like technical AI safety research, moral philosophy, accidental nuclear war, and climate change. The co-founder of effective altruism, Prof Will MacAskill, who many of you will be familiar with from episode 17, was recently on the show discussing moral uncertainty and its relevance to AI Alignment.

You can subscribe to that one by searching for Future of Life in your podcasting app.

OK, back to the interview with Dr Rachel Glennerster.

As I said, Rachel is now Chief Economist at the UK’s aid agency, DFID, and before that spent many years leading the famous research institute known as J-PAL, based in the MIT Economics Department.

Her research has included randomized evaluations of community-driven development in Sierra Leone, empowerment of adolescent girls in Bangladesh, and health, education, and microfinance in India.

Rachel also happens to have been one of the founding members of Giving What We Can, pledging 10% of her income to the most effective charities she could find.

So now I bring you Rachel Glennerster, interviewed first by Nathan Labenz, and then by me.

—

Nathan Labenz: Thank you for being here. Very excited to have this conversation. What are you working on at the moment, and why do you think it’s especially important?

Rachel Glennerster: I’m doing lots of different work at DFID But let me talk a little bit about some research work I’m doing, which is evaluating a mass media campaign, a radio campaign run by development media International. Which is an NGO here in the UK. They are doing a family planning program on radio in Burkina Faso. I think it’s really important to look at Radio and mass media, because it’s a very cheap way to reach a large number of people, and you can make sure that the message is accurate and consistent.

Rachel Glennerster: The problem is that it’s very hard to evaluate radio for exactly those reasons, because one radio program reaches millions of people at the same time. So, it’s very hard to randomize. Now it happens at Burkina Faso is one of the few places in the world where one can evaluate this effectively. It’s also true that at DFID we’re learning a lot about the Sahel. It’s an area that hasn’t had a lot of UK interest until recently. Also family planning is a hugely important issue. Because if you get the demographic transition right, it can have incredible benefits for women, for economic development, and for the health of children.

Nathan Labenz: So, why is it that is more testable in Burkina Faso? Is that a language group issue?

Rachel Glennerster: It’s a complex series of factors that means that you have a lot of different radio stations across Burkina Faso, and that indeed have different languages. So, there’s less spillover. So, you can randomize at the level of the radio station. You also have people who are so poor that they can’t afford radios. We’re randomly handing out radios to women who don’t have radios. So, we have two levels of randomization. Both at the radio station, and within a radio area some women already have radios, some women are given radios and some aren’t. So, it’s a conglomeration of things that allow you to be able to test.

Trajectory of aid and development work

Nathan Labenz: How would you sketch the intellectual trajectory of aid and development work over the last three decades?

Rachel Glennerster: I think it’s worth looking at two different trajectories. There’s the trajectory of the aid sector or the development sector, and then there’s the trajectory of the academic research on development. Those are a bit different. One of the nice things about RCTs is they’ve brought those two really closely together. I think the trajectory of the aid sector has been one in which there was not enough emphasis in my view about understanding really rigorously what works.

Rachel Glennerster: There was a lot of different theories and views about what we should be doing, and you see these big swings and fads between the view that development was mainly about investment in physical capital. The reason for aid is just that countries don’t have enough investment, and so we should be building stuff. Places like India had five year plans and they built steel factories. And then there was a big swing saying, no, we need to worry about human capital and not just steel and because we can grow but people aren’t benefiting, people they’re still malnourished, they’re not learning.

Rachel Glennerster: You see these big swings and interest about what we should be doing for development. Not a lot of it was very based on data. There has been a really big change in the last 10 years or so especially within DFID and within the World Bank to really seriously think about what is the evidence behind the decisions that we’re making. One of the reasons I moved to DFID is it has been one of the aid agencies that has changed the most to constantly be questioning ourselves. We’re going through that whole process at the moment of looking at what we’re doing and saying, is it really evidence based? What’s the new research say? What stuff should we stop doing? Because the new evidence is saying we shouldn’t be doing that.

Rachel Glennerster: That really is an important we see changing in the aid world. Within academia where most of the RCTs are happening, I think RCTs is a part of a much bigger change, which is about, again, thinking carefully about what’s the causal effect? There’s a lot of work on descriptively what’s happening in developing countries, which I think is really important. There was quite a lot of focus on ideology, I think, which I think there’s been some move away from. But to be honest, just not a lot of work happening on development.

Rachel Glennerster: One of the biggest changes in research in academia in economics, it’s just there are a lot more people thinking about the developing world, and that’s great. A lot of them are doing RCTs but if you look at the data, actually, there’s just as much non RCT workers there as it was before. There’s now more RCT work and there’s just more people. Development was a bit off the side. People thought about different issues. What people have realized is the questions in development are actually very similar to the questions in other bits of economics, and we should be learning from each other.

Rachel Glennerster: You’ve seen things that have been discovered in the developing world, like, a lot of behavioral economics lessons came from development, and they’re now being taken up and used and learnt from in rich countries. I think that better integration of development into understanding that we’re all the same and there are a lot of similarities and a lot to learn. I think that’s been really important to see that lessons going in both directions.

RCTs

Nathan Labenz: Moving to RCT’s in general and the state of debate around how much we should rely upon them. You mentioned that it’s kind of a 50/50 split right now, in today’s work. Do you think that’s an appropriate split? Do you think that it should be all RCT’s? What do you think is the right balance as we try to figure out what is obviously a very complicated world?

Rachel Glennerster: I think it’s really important to say that all of us who have worked on randomized trials have never suggested that this is the only methodology that you should use. Sometimes it’s held up as a straw person that we go around saying, this is the only methodology, people criticize us for saying it’s the only methodology, but nobody who’s done RCTs has ever thought that that’s the right approach. I think the right way to see things is you have a toolbox of ways to answer questions, and the right tool depends on the question that you’re asking.

Rachel Glennerster: I think we need good descriptive work to understand what the problems are. A lot of development programs just fail because they’re trying to solve a problem that doesn’t exist. They’re just solving the wrong problem. The first really important thing you’ve got to do is really understand what the issue is in any given area. If we’re worried about girls not going to school because of menstruation, well, let’s start by finding out whether they actually don’t go to school more when they’re menstruating. That’s a really basic, obvious thing. But we actually need more work on that kind of understanding the context, understanding the problem, is really important first step.

Rachel Glennerster: When I started doing agricultural work in Sierra Leone, and the first thing we did was work with the government to do a really detailed analysis of what are the problems for smallholder farmers in Sierra Leone? Not RCT, just descriptive. It turned up all sorts of interesting facts that people weren’t aware of. I think that’s really important, I think, then doing an RCT is useful for answering a really specific problem, a really specific question. But I think the best RCTs are the ones that test a theory. They test something that’s more generalizable than just does this program work? Its asking a question about human beings.

Rachel Glennerster: Here is an example. I did a project looking at how to improve immunization rates in India, which was fantastically effective. It started with a first assessment of what are the health problems in this area? Only 3% of kids in this part of India were getting fully immunized. Given that immunization is one of the most effective things that you could do, that rate is just appallingly low. There were a number of theories about why that could be, and a lot of people said, “Well, people don’t trust the doctors, they don’t …” Well, not doctors because you rarely get doctors in rural India or rural anywhere, but nurses and clinics. So, they don’t trust the formal health system.

Rachel Glennerster: There was also a question of, so the clinics are often closed, so is that the problem? Is it that when you go and take your kid to the clinic, it’s often closed? Is it nurse absenteeism that’s the problem? Or is it just a behavioral economics thing that you’re happy to get your kid immunized, but you’ll do it tomorrow?

Rachel Glennerster: We read all this behavioral economics and we said, “Well, maybe we should look at that.” But we also wanted to test these other ideas. One arm made sure that without fail, there was someone to immunize your child and another arm did that, but also provide a small incentive. So, yes, we were testing a program but we were also asking a more fundamental question, which is, why don’t people get their kids immunized?

Rachel Glennerster: What we saw in the data is a lot of people got their kid immunized with one immunization, but they failed to persist to the end of the schedule. Which already, that’s just descriptive data and it starts to tell you, it’s not that they distrust the system or that they think that immunizations are evil, because they’re getting their kid one immunization. It’s more question of persistence. Now, fixing the supply problem increased the number of people getting the first shot, and the second shot, but again, it failed to fix this persistence problem. Where the incentive effect worked, was it helped people persist to the end.

Rachel Glennerster: That tells you that one of the big problems was this persistence problem. It tells you a lot about why immunization isn’t happening. Now, that project was completely impossible to scale. We were handing out lentils and the middle of Rajasthan where nobody showed up. It was just you would never … This was like economist designing logistics. It was a disaster. We learned a lot but you would never want to actually do a program like this.

Rachel Glennerster: A colleague of mine did something similar, where another program we had just done where we ended up improving teachers attendance by having cameras that were wrapped in sellotape and signed. Again, the logistics was a nightmare but it tested a theory. Once you have that, you can think about what’s the implementation issues? How do we implement this at scale because you better understand the problem. You want to use an RCT when you can test a specific problem and get an answer to why something is an issue. It’s an important question, you can answer it well, and it has broader implications. But you also need to use other types of methodology when your question is of a different kind.

Small vs. society-wide capability building

Nathan Labenz: Tyler Cowen, who’s a recent guest on 80,000 Hours podcast, just published a book in which he basically argues that the number one focus should be on economic growth; there’s where all of the good that we enjoy comes from, subject to some constraints based on general human rights. I think he has a pretty intuitive approach to what exactly that definition is. But it seems like you would broadly agree with the emphasis on growth. At the same time, some have argued that the focus on RCTs and what has been called the aid effectiveness craze, quote unquote, is focusing our attention on small issues that maybe are distracting from the bigger questions of broad economic growth and progress in these societies. Do you think that is a valid worry, and how do you square those two levels of very small and society-wide capability building?

Rachel Glennerster: I think there’s a number of different things going on here. One, I first need to object to, which is the characterization of RCTs as aid effectiveness. Most of the work on RCTs has not been focused on aid. I think it’s really important to understand this difference, which is most of the money that goes into poverty relief is money spent by people in developing countries, governments and individuals. And actually, if you look at most of the people doing RCTs, they don’t think that their audience is aid donors. Their audience is the government of India and the government of Brazil and the government of Indonesia, and to some extent big companies or other individuals there. Because that’s where the money is, to be honest. Let’s just remember, there’s aid and there’s development, and aid is only ever one small part of development.

Rachel Glennerster: I do agree that improving the policies of developing country governments is a hugely important way to impact local poverty. So the RCT craze is not about aid effectiveness, it’s about government effectiveness, poverty effectiveness. So that’s one slight quibble. Then there’s the heart of your question, which is policy versus working on small questions. Do RCTs work on small questions, and then there’s how do we think about development versus, say, working on improving someone’s health or education now? Again, those are I think two different questions.

Rachel Glennerster: I think actually RCTs should not be seen as looking at testing this specific program, they should be seen as testing big questions that can then influence policy. For example, you might test a specific project on education. A lot of the work on education has suggested that the most effective thing we can do in education is to focus on the learning within the classroom. It’s not about more money, it’s not about more textbooks, it’s not about … And that’s what governments spend their money on. They spend it on teachers and textbooks, mainly teachers. But more teachers doesn’t actually improve learning. More textbooks doesn’t improve learning. But that’s what the Indian government is spending their money on. So if I want to help the Indian government on education, I want to test those different things about how the Indian government could improve their education, and then help them reform the education system. And what this set of RCTs has suggested is, not just that it’s about the pedagogy, but it’s specifically about the problem that the Indian curriculum is up here, and most kids are here.

Rachel Glennerster: If you look at the data, just descriptive data, again, the power of descriptive data … within an average Indian classroom in 9th grade, none of the kids are even close to the 9th grade curriculum. They’re testing at somewhere between 2nd grade and 6th grade. No wonder they’re not learning very much, ’cause the teacher, the only thing that a teacher has to do by law in India is complete the curriculum, even if the kids have no idea what they’re talking about. So yes, you have RCTs testing very specific interventions; all of the ones that worked were ones that got the teaching down from the 9th grade curriculum to a level that the kids could actually understand. Now the lesson from that, the big lesson for the Indian government if they were ever to agree to this, is change your curriculum. That’s the biggest thing that you could do. Reform the curriculum and make it more appropriate to what children are doing. So yes, you’re testing little things, but you’re coming out with big answers.

Rachel Glennerster: Now the final part, and I think the hardest part, is economic development versus, say, working on health and education. At DFID, we have shifted a lot of emphasis relatively recently into trying to do more on economic transformation, under the recognition that the biggest reductions in poverty as you say, have come from big transformations in economic policy. So the big opening up of India and China towards more market-oriented economies … And I’m not saying markets solve everything; they absolutely don’t, but when you’ve got a system as screwed up as Communist China, making prices have some influence moves you an awful long way, and can really help transform the economy. And the same happened in India, and you saw massive reductions in poverty, by just a move towards a slightly more sensible economic policy.

Rachel Glennerster: When I was recently doing my ranking of what are most effective things that DFID could do, we were saying, “Well, if there are cases of countries that are as screwed up as China, helping them move to a more effective economic management, that’s gotta be the most effective thing that we could for poverty. You can’t do that as an outside donor, unless someone’s willing to do it. So where you see … I would say Ethiopia at the moment is going through a tremendous reform, and we really ought to be focusing attention, and helping Ethiopia in that transition. Tremendous potential, because they’re absolutely fundamentally changing policy there in ways that could be really beneficial to the poor. So jump on those opportunities, but you can’t really make them happen. It’s something that the developing country themselves has to decide to do, then help them as much as you can.

Rachel Glennerster: Then there’s the question of what do you do to promote economic development in countries that aren’t going through this fundamental reform process. You can nudge them a bit in the right way, you can maybe help improve trade policy, you can help reduce trade barriers … there’s things that you can do. But in a lot of countries it’s not entirely obvious what you can do to promote economic development. We need a lot more research, a lot more understanding about how to do that, because I absolutely agree that it’s fundamental. But we don’t always have all the tools that we need to make economic transformation happen.

Rachel Glennerster: Now, think about our own economies, and our own … It’s not that we only ever worry about economic development. We also worry about health and education, because we don’t grow in order to have more money, we grow to have better lives. So we want to make sure that that money translates into actually better lives. So we need to be thinking about poverty relief domestically within developing countries’ health and education.

Rachel Glennerster: And we know a lot about how to do that well, so we need both to take the opportunities for economic development and growth when we can, and really come in there heavily where there’s an opportunity, but we also need to be working on health and education, not least because we know that those things are really important for economic development. We know that there’s high productivity improvements if kids are given the right nutrition early on; there’s about a 10% return to investing in education. To some extent, you can’t have economic transition without the building blocks of human capital. In the classic economic growth model, there’s human capital, people and physical capital. If you want growth, you need to be working on all of those things.

Overrated, underrated

Nathan Labenz: I wouldn’t be doing my job if I didn’t get to the overrated, underrated quick hit list.

Tyler Cowen: I’m Tyler Cowen, and I approve this use of Overrated and Underrated.

Nathan Labenz: …And so I’ll give you a number of prompts, you and you can respond with “overrated” or “underrated”, and of course you’re free to pass on any of them if you don’t have a strong view, or if you’d rather just avoid the topic.

Nathan Labenz: The first one, overrated, underrated: charter cities as a means of promoting the sort of growth that we’re talking about.

Rachel Glennerster: I’m not a fan of charter cities, but I don’t think anyone else is either, apart from one Nobel Prize winner.

Nathan Labenz: How about going along to get along with your colleagues?

Rachel Glennerster: I think it’s really important to learn how to influence and how to get along with your colleagues if you’re going to make change, so underrated.

Nathan Labenz: Starting a business in the developing world.

Rachel Glennerster: Probably underrated. Social entrepreneurship: overrated. Business: underrated.

Nathan Labenz: And how would you draw a line between those?

Rachel Glennerster: Social entrepreneurship is small things that you … developing a solar torch to sell to people. Many of those do not take off in a big way, partly because people don’t have a lot of money, and I think you could have a much bigger impact by working in big organizations. I do think that there’s a lot of evidence that businesses in the developing world are really badly managed, and there’s a lot of improvements that could be made. Basically people want jobs; they don’t want money to create their own businesses, they want jobs. So managing to get good, effective private sector businesses working in these countries is really important. Now, whether this audience is well placed to run those companies is a question, but I know people who’ve spent many years in development, working as NGOs or working as RAs for me, who set up businesses. And I think that’s great.

Nathan Labenz: How about cash transfers?

Rachel Glennerster: Cash transfers I think were very high, appropriately. People have been a little bit down on them recently because of some recent work saying maybe the long-term impacts of one of them wasn’t as much as people had hoped, but I think when you look at the literature as a whole, it’s very positive, including on the long-term benefits. And even if the control group catches up, getting people out of poverty earlier is still really beneficial. So, again, it’s underrated.

Nathan Labenz: Once was appropriately rated. now underrated.

Rachel Glennerster: Yeah, exactly. Was appropriately rated, now perhaps underrated.

Nathan Labenz: Okay. How about gene drives for mosquito and other disease vector control?

Rachel Glennerster: Okay, I’m going to pass on that. I don’t know enough about that.

Nathan Labenz: Maybe this one as well. Genetically modified and CRISPR crops?

Rachel Glennerster: So don’t know about CRISPR crops, but I’m a big fan of GM crops, and particularly just improved agricultural varieties in the developing world. Hugely beneficial. Now, some of that you can get without GM, but I think we’re probably a little bit paranoid about GM.

Nathan Labenz: How about cracking down on tax havens or other sources of illicit financial flows?

Rachel Glennerster: Underrated. We should do more of that.

Nathan Labenz: What’s the mechanism by which that benefits everyone?

Rachel Glennerster: A huge amount of money flows out of developing countries into tax havens. It’s not really about developed world tax havens; it’s also a big problem in terms of fueling corruption. We in the West … there’s a big opportunity for exposing the bad deals that are done with bad governments in developing countries that are often done in the West. We have a huge culpability for that, and we ought to be doing more to stop it, and I’m pleased to say that DFID is doing more work on that.

Nathan Labenz: Micronutrient supplementation.

Rachel Glennerster: Micronutrients, underrated. Supplementation, we still need more work on that, because it’s … The way we’re putting micronutrients out at the moment doesn’t seem to be working very well. Anemia is probably underrated. There’s a huge, huge problem; it really affects productivity, it affects cognition, it’s a major problem. We haven’t quite figured out how to address it though, so problem is underrated, solution needs a lot more work.

Nathan Labenz: Reforming developing country macroeconomic policy.

Rachel Glennerster: Yeah, so macroeconomic policy is really important, but we’ve actually kind of figured it out. If you look at inflation, it used to be a major problem. When I was doing development economics, half of the course was about how to deal with hyperinflation. Virtually nobody has hyperinflation any more. It’s a real major success that we don’t talk about enough. So it’s really important, but if you’re suspicious, I shouldn’t say this, but we’ve kind of nailed it.

Nathan Labenz: Okay. Couple more. How about pre-registration?

Rachel Glennerster: I think that’s overrated. And that’s a little funny coming from me, ’cause I wrote one of the papers saying that we ought to do more of it in economics. I’m now finding some of the downsides. Yeah, there’s a big move to pre-register in advance what your analysis is going to be, and sometimes tying your hands is not actually a good idea. So pre-registering that you’re doing a study is really important, pre-analysis plans, which say. “This is exactly how I’m going to analyze my data when it comes out,” can be a problem because when you look at the data and stuff has happened in between, it may be really important to change how you do the analysis. Plus, they’ve found referees hate it. That’s probably less of an issue for this audience.

Nathan Labenz: Do you think that people are not doing that extended analysis, or that it’s being unfairly dismissed as a result of the pre-registration?

Rachel Glennerster: I understand that people worry that you run a trial, and then you test it, test your results on 50 different outcomes, and you promote the one that had a positive effect. Most academic work doesn’t work quite like that, because you would never … Your referees force you to show 50 robustness checks, and you don’t get past if only one of them had a positive effect. I think we need to rely a bit more on theory. Theory tells you which things should go together, which things should be important, and I think theory can be as an effective way of looking at the data and pulling out patterns, and is a bit of tying your hands … and might be a more effective way of tying your hand than pre-analysis plans.

Rachel Glennerster: I’m not saying you should never do them, they’re not the simple answer that people thought they were.

Nathan Labenz: Okay. How about reading the news or the newspaper?

Rachel Glennerster: Reading the news of the kind that you already support, we should be doing less of. Reading things that shock you out of your … that comes from a different perspective, I think is … We don’t do enough of. I try and read about African politics, for example. It’s not on our news normally, but it’s really interesting. Reading the news from serious people who cover African politics, I find really helpful in bringing a different perspective. And my son is brilliant at saying, kind of pushing to say, “Read what people of different political opinion are saying, just to keep your horizons broader.” But I think often when we read the news, we read the things that confirm what we already know, and that’s not very helpful.

Robert Wiblin: OK, thanks so much to Nathan doing such a great job filling in for me with no notice. We pick up now with me speaking with Rachel three days later at her office in London.

Generalizability

Robert Wiblin: Earlier this year, I spoke with Eva Vivalt about generalizability of our [inaudible 00:57:53]. And she had this archetypal stylized fact that at least in her sample, the typical result differs from the average effect found in similar studies so far by about 100%. Which is to say that if all existing studies of an education program find that it improves test scores by 0.5 standard deviations, the next result in her sample was as likely to be negative or greater than one standard deviation as it is to be between zero and one standard deviations. I guess … do you think that Eva is potentially overstating the implications that this has, and how dire the situation is for external validity of our [inaudible 00:58:25]? What’s your general perspective on this?

Rachel Glennerster: Yeah, so I think it’s really important to distinguish the different causes of why a different study might have a different result. Because we take away different conclusions, we act differently depending on the reason for why a different study might have a different result. So one reason why a second study might have a different result is the problem didn’t exist there. So then I’m not at all surprised that you have a different finding in another study. It doesn’t worry me at all.

Rachel Glennerster: A second reason why you might have a different effect in the second study is it’s implemented less well. In the first study, you had 80% take-up, and the second study, you had 20% take-up, right? So again, you don’t want to just compare the results of the two studies, you would then want to adjust for take-up in those contexts.

Rachel Glennerster: And the third reason is that people behave really differently in different contexts. And that’s the … that in a sense is the assumption behind saying this is a problem. And I guess my reading of the evidence is that actually, most of the variation between studies that we see is either they’re actually implementing a completely different kind of program, and we don’t have enough studies so we bundle a whole bunch of things that are completely different together, and it’s not wonder we get different results, or the implementation was really different and bad or really good. And that just tells us we really need to work on implementation. It might tell us really important things about-

Rachel Glennerster: … really work on implementation. It might tell us really important things about, this is a really hard thing to implement, and that’s a really useful lesson. It’s not really about generalizability. Generalizability for me means, people act differently when they’ve got faced the same problem and they’re given the same incentives, but they respond to those incentives really differently. And my reading of the RCT evidence is that, actually we get surprisingly similar results if anything across different studies.

Robert Wiblin: So you have this really nice article in SSIR, which we’ll put up a link to which readers can read, which kind of lays out your view and gives like what’s the first specific cases. It seems like the implicit claim here is that, if you were able to see how well the replication was being implemented and you knew something about the situation, then you will be able to predict ahead of time, like whether you would expect to get the similar result as in previous studies on or not.

Robert Wiblin: Do you know of any evidence of people who have actually tried to do that, to run an experiment where they’ve tried to see whether they can predict ahead of time? Because I mean, the fact that they were these trials being run that got like negative or very different results suggests that, at least some of the time people couldn’t predict that it wasn’t going to work in this other context, at least not ahead of doing the experiment.

Rachel Glennerster: So there’s two parts to that question. The first is, is there a methodology we can use? And yes, absolutely, there is, and we haven’t used it so much, but in a recent study I have done, just maybe not exactly what you’re talking about, but in a recent study I’ve done, we asked for priors at the beginning. Actually before we got the results, we asked different kinds of people what they expected the result to be.

Rachel Glennerster: In this case it was a longterm followup. So people were shown the original result and then we say, “What do you think the result would be, in five years time?” But other people have done it. Stefano DellaVigna has done a nice study where he sort of asked people, what you would expect to find in this kind of study, and shows who’s good at predicting and what people are good at predicting.

Rachel Glennerster: [inaudible 01:02:02] he asked people what you expect to find and then he runs the study. I think we should be doing a lot more of that. I’m hoping that we will start doing it. We did it as a response to Stefano’s paper and so hopefully other people will also pick that up. Because one off that’s not enough, but if we get enough studies, that’s quite useful.

Rachel Glennerster: The other thing is, do we know whether something will be implemented well beforehand, and wouldn’t we to know that. So it’s quite interesting summit. So some of the psychology literature actually shows that, there’s a lot of information in people’s heads about what things were replicated and what won’t. And it’s not in the paper necessarily, but if you ask experts they’re like, that study will replicate and that won’t, and they’re quite good at predicting.

Robert Wiblin: There’s a beautiful replication piece in science that came out a few months ago, which show the psychologists were extremely good at predicting which papers were actually gonna replicate and which weren’t, even though they were all published in the same quality journals.

Rachel Glennerster: Right. So somehow I think that does suggest that the experts, there’s some information that somehow isn’t in the papers about, that people understand whether this thing is replicable or not. I don’t take exactly the same conclusion as you do that people aren’t good at knowing whether something is going to work. I think often we’re doing trials of risky things in risky places. Part of what we’re trying to do is work through the logistics of trying to design. So trying to design things which are then easy to scale up.

Rachel Glennerster: So the first step is to test whether something works when it’s implemented extremely well, and then the next step is, can I reproduce that result in a more scalable version? That didn’t work, so let’s try it another way. So it’s kind of a process where at the end we’re kind of iterating onto something that should be more consistently able to work at scale. My slight concern with what Eva does is, she sort of throws all of those things into one pot and kind of runs a regression.

Rachel Glennerster: And when those different studies were trying to do really different things, some of them were trying to say, how would this work if you did it extremely well, with the best implementer, and you paid really close attention and the other one is, would this work if we had a crap implementer?

Rachel Glennerster: So yes, in those cases you would expect it to be really different. And often that’s exactly the question that you’re going to test. And it’s not that it’s a failure, it’s a learning. Like you’ve learned that, no, this kind of implementer can’t do it. That doesn’t mean the first study was wrong. It means, we’ve got to try another implementation strategy.

Robert Wiblin: Do you agree that Eva’s results are at least to show that if you were going to be, like you can’t really do that very naive thing of just looking at the average effect size across a bunch of studies and then saying, well that, like if I run this again somewhere else, like without even being an expert in how to run this program, it will have a similar outcome, that it does show that, that would be an error of judgment. Nobody is silly enough to do that.

Rachel Glennerster: Well, this is one of the reasons why I think we have to be quite careful when we do meta analyses, because some of them are just kind of throwing everything together, and you’ve got different levels of take up, and it’s actually the objective of the study wasn’t really to test that thing. And so, you need quite a lot of judgment, more judgment to do a good meta analysis than I think people realize.

Robert Wiblin: So, it seems like often with an intervention there aren’t that many different studies, testing how well it works in different places. And it seems like you might then be kind of underpowered to figure out what the contextual factors are that the matter at least using that method, it seems like you’re going to have to use a lot of judgment based on your experience with the crown of what things are likely to matter, and also just what prior is about, what things are likely to matter in different places. Is that right?

Rachel Glennerster: So I think the way we should think about this is, we look at these studies and this again is set out in more detail in my SSIR paper, but let me try and summarize that, which is, we have very few studies that test exactly the same thing in lots of different contexts. We have quite a lot of studies testing some fundamental underlying principle about human behavior. The trick is then to take that fundamental principle about human behavior, which we’ve tested many times and now know to be true, and think about how to implement that in the local context.

Rachel Glennerster: And that could be about, offering lentils in one context, and ice cream in another, whatever it is. But those things don’t need to be tested by RCTs. Some of that is about, it is knowing your context, but it’s just, it’s basic logistics, and some of the basic logistics we do need to test every single time, but we don’t test with an RCT, where you test with good monitoring processes.

Rachel Glennerster: So it’s like saying, I know there’d been enough studies that I know if I hang a bed net it will work. I’m not going to test that again. I’m going to test-

Robert Wiblin: Did they hang them up here?

Rachel Glennerster: … did they hang them up?

Robert Wiblin: Which is probably a lot easier to test.

Rachel Glennerster: Much, much easier to text. So it’s doing that causal pathway and saying, which bits are making sure that you’ve analyzed the problem right, that they actually have malaria? The next step is, if you get the people have tested is if you give malaria bed nets for free, normally you get higher take up and people are most likely to hang them, but the third thing is, how do we make sure that we have a monitoring system that the malaria bed nets actually get out to people, and also maybe test occasionally that they’re hanging them up. But that doesn’t need an RCT, that’s good monitoring.

Robert Wiblin: If you need a lot of trials to kind of establish a stylist fact about how humans behave, how many things have we learned of that kind? It seems like it might be a fairly short list if you needed like dozens of studies to figure out that something just recurs like in most cultures, and most countries, and most situations. Is they kind of a list of like these things that we’ve learned, these underlying principles that we use whenever we develop any program at all?

Rachel Glennerster: Well maybe not every program, but we know that people are very sensitive to pricing convenience in the take-up of healthcare. So never charged for preventative health care. Like that’s a pretty big dome, policy conclusion, but we go a lot of different studies that point towards that. So in the health field there are a number around this, price and convenience type of things.

Rachel Glennerster: And then in education I think we’ve learned this more general lesson from many, many studies that it’s really about, it’s not about the inputs, is about the teaching, how people teach and the biggest problem in most developing countries is that, the teaching is way above the heads of most of the kids. So they’re on grade two, you’re teaching them at grade six. That’s a pretty fundamental thing, which we put a lot of progress, a lot. That’s going to come into a lot of different education programs to use that principle. People respond to incentives.

Robert Wiblin: The price goes up, people buy less so.

Rachel Glennerster: So there’s quite a lot of, I’m not going to remember all of them off the top of my head, to do a plug for my old organization, [inaudible 01:09:34]. The last thing I did before I left was bang heads together of the various academics working in an area, and they of course they love to kind of put caveats on everything but say, what are the main crosscutting principles that have come out of RCTs in your area?

Rachel Glennerster: So those are called, policy insight notices and you can go to [inaudible 01:09:57] and under each one they will at least list three, like common crosscutting principles that they found. So in agriculture, and in health, in education, in governance. So that will give you a list of some of those. So it’s, for most of them it’s not more than about three, but they are quite general.

Robert Wiblin: This potentially like makes RCTs look like substantially better value if you can come up with these principles they’re gonna apply to basically every other program in the area in future because, [crosstalk 01:10:25]

Rachel Glennerster: Not every other program, but …

Robert Wiblin: But at least something you might want to consider in most cases.

Rachel Glennerster: Yes.

Robert Wiblin: Because I guess sometimes people worry that, each RCT is quite expensive, and then to test all of the programs that we might consider ever running, it’s just prohibitively expensive. But if you can find, these underlying principles and it’s like of much broader value.

Rachel Glennerster: So I think that’s a point that people really, really get confused about which is, the more academic and abstract the RCT in some ways the more policy relevant it is. Which sounds really odd and counter intuitive, but the point is, an RCTs that tries to test one of these fundamental principles of human behavior, is actually much more useful, because if you’re just testing a package of things and you can’t really tell an underline principle from it, you’re testing whether these six things together have an effect.

Rachel Glennerster: You can’t take that to another context. You can’t learn as much from it as if you go in deliberately saying, my question is not does this work, but are people price sensitive in this area? Do people use something more if they pay for it? Like I’m testing this more general principle that’s more generalizable and it’s better value for money.

Robert Wiblin: So there’s a lot of different potential sources of heterogeneity, like differences in culture or like differences in the quality of implementation or the fact that a program was different in some ways that like aren’t really getting picked up in this metro analysis, and also just like random noise of course. Do you think that any of these is like underrated as how significant it is or how much variability it creates?

Rachel Glennerster: So I think implementation quality is something that people don’t take into account enough, but that there is by the kind of program. I think what we ought to be doing is looking, if someone is taking a deworming pill, there’s not that much difference in the quality of the pill usually. Right? The quality is very easy to measure in that did someone take it. But if you’re talking about a training or graduation program, then it really matters. So I think we should be looking at, when we do these metro analysis using our theory to say, is this something where the quality is going to matter? Quality of implementation is going to matter or is it not?

Robert Wiblin: Do you think we potentially have delivered too many programs where the quality of implementation matters? Like I guess that can potentially reduce the expected value quite a lot just because there’s a high chance you’re gonna screw it up.

Rachel Glennerster: So I’m a big fan of, in general unless you got a really good evidence against this of, doing less complicated programs, fewer components, and just doing one thing well massively, like I think that’s a huge problem that we try lots and lots and lots of different things, and I don’t mean task lots of things because, we want to test lots of things to find the one really good cost effective thing, and then we should scale that up massively. And we don’t do that enough, and that’s one of the things I’ve been saying [inaudible 01:13:27] by the way.

Robert Wiblin: So I saw this paper that came out last year which said that about 95% of papers published in development economics were empirical. Do you think that that’s potentially too high, given that it sounds like you think there’s a substantial role for like people having theories about, what things work and how people behave. Maybe like the empirical revolution has gone like a little bit too far?

Rachel Glennerster: So actually what I think is happening, so if I remember that paper right, they count as empirical anything that has an empirical element in it.

Robert Wiblin: So it’s like too general or very expansive definition?

Rachel Glennerster: What you see a lot, and I suffered from this myself because I don’t do theory in the sense of like in a paper theory is math. It’s a mathematical model. I’ve been using theory to mean kind of, thinking about general principles, and both are true. Like a mathematical model is just a way of formalizing those general principles. What you see, what the trend has been is to neither have purely theory papers or purely empirical papers.

Rachel Glennerster: What you’re seeing a lot more of is papers that have both theory and empirics, and that I think is great even though it makes my life harder at the moment. I’m in the process of adding mathematical models to what were purely empirical papers. But it’s really pushing me to understand my empirics better, and realizing that our results in one of our papers was actually really quite counter intuitive and against all the other models out there where it felt kind of like [inaudible 01:15:01] intuitive result when we first got it.

Rachel Glennerster: But then when somebody’s … when a reviewer said, no, I want to see a model behind this, we realized that it actually was countered to the standard models in the literature. So I think that’s exactly pushes us to do what I’ve been saying which is, go backwards and forwards between the empirical and the theory, and that used to be an economics that would happen, but one set of people would write the theory and another set of people would do the empirics and hopefully they were going backwards and forwards, but it wasn’t in the same paper. So the trend you’re seeing I think is less theory on its own and more kind of combined papers. And I think that’s great.

Robert Wiblin: Just coming back to the question of whether people can predict, which program is gonna work in other situations. So when that science paper came out with a 20 or was it 21 social science paper and [inaudible 01:15:52] predict. I actually met a quiz on our website where people could test their inability to predict whether the papers would replicate.

Robert Wiblin: This was because after the replications had come out, but most people will be like, they haven’t read all the details, even if it read the headline, they don’t know which ones. So I wonder whether it’s possible to go back retrospectively, I’m like, once you’ve done like replications, even after the fact you can go back and see whether people can predict now which ones did replicate in the past, or like how much things are transferred from one country to another.

Rachel Glennerster: The thing is in economics we just don’t have enough like actual pure replications. They’re not replications in the way that that article or you’re talking was, which was literally running the same thing. We just don’t do that. We tend to be doing things that also predict that general theory, but it’s done in a very different way. And then you could say, well, yes, but then you can always reinterpret what you find and say, well, it didn’t fit that theory because it wasn’t implemented well or it wasn’t actually the theory was different. But that’s right.

Rachel Glennerster: I mean that we changed the theory. So I think you can’t do what they did in economics because there’s literally there’s micro-credit and graduation program, and now there’s one on politics which I [inaudible 01:17:04] not happy with, but there’s kind of three sets where we’ve tried to do the same thing everywhere, and I’m not sure that they always make sense to do that.

Robert Wiblin: What about just trying to predict the level of external validity from one country to another, or one implementing organization to another implementing organization, not an exact replication. We’re predicting, how much will the impact to transfer from one case to another. Is that something that can be done?

Rachel Glennerster: I mean, that assumes that you’re going to do it in the same way.

Robert Wiblin: No, but I’m saying do it differently, but just say, well, here’s the result of this other thing. We’re going to try to do it in this other case. It’s going to be different in these ways, and also it’s a different country and the other people doing it, maybe they’re like, no it’s competent or something like that. Like, what do you think the impact will be on average, of this new version?

Rachel Glennerster: I think it’d be really, as I say, I think it’d be really interesting if people wrote down their priors beforehand, the researchers and the implementers, and it said, I predict that this is going to be the impact and if it’s not, these are the things that are going to matter. Because you’re not going to know whether there’s a hurricane or whatever, but the things that are gonna matter are, take up.

Rachel Glennerster: I think take up is going to be good, but this is really gonna, my impact is going to be much less if take up is less or if absenteeism is high or what. These are the three things that I think are the potential risks to this not having an impact. So I think I would do both. What’s the prediction here? And these are the things that, that prediction, these are the assumptions behind my prediction that take up will be high and this will be. So if these don’t follow through, that’s why I wouldn’t get the result.

Robert Wiblin: Really there’s something with the quiz of the social science replication because we put an assignment was that we found that, just total random people on the internet were equally as good at predicting the replicability of papers as like psychology professors were.

Rachel Glennerster: Really?

Robert Wiblin: When you aggregated them both. Yes. So I was quite surprised by that because I tend to be like a kind of a pro expert person. But in this case it’s the same. I mean, I don’t expect that this will transfer over to development and economics as much because, in this case often it was like people looking at me being like, here’s this primary experiment, smells like nonsense. Like I don’t really believe that. And like both, like a normal person and a psychology professor can both see this.

Robert Wiblin: In this case there might have been actually be more expertise hopefully. I actually mean hopefully not in a sense, because if you could just take random people and survey them on whether they think this experiment will replicate, then that’s ideal. It’s a lot cheaper than running it again. And even better if they can also anticipate like where it would generalize to and where it won’t. I guess it’s something that I would love to see more papers on that.

Rachel Glennerster: And another thing if we got more priors, we could start using more Bayesian statistics, which will be great.

Robert Wiblin: Do you want to paint a vision of how that might work?

Rachel Glennerster: No, but I am following closely some, good economists who were thinking about this problem. When I last heard someone present on how we should be shifting to Bayesian statistics and economics, they said, well the minor problem is, we can’t put confidence intervals on it, and I’m like, until you can put confidence intervals, we can’t use it. We can’t use it, we won’t get published. Because we can’t say whether the result. If you can’t tell us how to say whether the result is significant, and they’re like, no, but that’s the whole point of Bayesian.

Robert Wiblin: Right. I see.

Rachel Glennerster: That it doesn’t …

Robert Wiblin: It doesn’t. They’re trying to shoehorn it into frequentest statistics.

Rachel Glennerster: Exactly. But I just don’t see how we’re ever going to get published if we can’t say something like that.

Robert Wiblin: You can’t do like a confidence interval on like the credences or something like that across people? I don’t know. I guess.

Rachel Glennerster: I don’t know. That was the result I was told.

Robert Wiblin: Interesting.

Rachel Glennerster: And so I think it might be well.

Robert Wiblin: I think we need some new bayesian journals.

Rachel Glennerster: Yes.

Robert Wiblin: All right, pushing onto a question that really interests me and might be a bit of an indulgence. I’m not sure how much listeners are going to care. But a year or two ago we tried to figure out the answer to this question of like, what fraction of social interventions work, because we’ve been using this quote from somebody his name I think David Anderson where he said like, most social interventions like when tested don’t work.

Robert Wiblin: And we found that like the more we thought about this, the more it really depends on exactly what you mean by work and exactly what you mean by social intervention as you might expect it. It’s like, we’re really going down the wormhole, but I ended up concluding that like probably quite a lot more interventions work in some sense like that on average you would expect them to have a positive impact.

Robert Wiblin: Maybe not in like every single implementation, but and maybe not like a very large effect, not Large enough to be detected in a small study, but like maybe even like more than half of the things you know have some positive impact in expectation. Do you have any [inaudible 01:21:33] general question of like, what should we say when we’re thinking what fraction of social interventions work, and just perhaps the reason why this matter is quite a bit is that, it’s relevant to what kind of a game you get from being empirical and doing running RCTs and measuring how well things work.

Robert Wiblin: Because it’s like, if one implement one intervention out of 100 had almost all of the impact of all of them, then you get a huge benefit for measuring. Whereas if like half of them work and they were all quite bunched together and how [inaudible 01:21:56] then maybe you should just save the money on measurement and just scale up more stuff.

Rachel Glennerster: So first of all I think it’s really, it’s quite hard to draw conclusions about what fraction of things work by what fraction of studies show positive effects. Because we tend to test the new things. So if some things we don’t test because we’ve tested them many times before, and we know they work, and so we don’t test them anymore. So bed nets would be an example of that. Like we don’t need to test the bed nets work anymore because that’s done.

Rachel Glennerster: So if you look at an existing supply of tasks now, hopefully not many of them are of bed nets because kind of that’s done, except that we’ve got new bed nets coming along. But just an example of something that we know works. So the flow of tasks tend to be the new things. So that’s one point. So you have to be a little bit careful about when you’re trying to answer that question of thinking about your sample selection of what is tested.

Rachel Glennerster: The second point is, I really don’t think it’s about, whether they work or not. It’s more your point about the bunching. Is it the case that most things will say, when they work, they work in similar amounts, and that I think is empirically false. So there’s some work that [inaudible 01:23:14] hopefully about to publish soon from looking at what USAID invested in innovation. And one of the problems in this kind of analysis is being able to look at all the things that were invested in, like to make sure that you don’t have a file drawer problem of tasks that were done and never published because they didn’t find something.

Rachel Glennerster: But you want to make sure that you have the whole portfolio of things of innovations that were funded, to be able to at kind of the range of outcomes. And what he finds from the work that he was involved in, in USAID is that they funded a lot of different things. The impact all comes from a very few, only a few massively scaled, and those are spectacularly effective and they reach a lot of people. And so I can’t remember what the total number is, 43 or 50 or something things were invested in.

Rachel Glennerster: But five pay for all of it. You get a fantastic return even if you assume that only five paid off. So when we’ve recently looked at [inaudible 01:24:26] when you look at cost effectiveness of things like on education, there were a ton of zeros, and there were a ton of zeros of things that, actually what we mainly spend our money on. So more teachers, more books, more inputs, more like smaller class sizes, at least in the developing world have no impact, and that’s where most of the money gets spent.

Rachel Glennerster: But the top ones, the most cost effective things are, 460 LAYS per $100 spent. So a LAY is the new educational equivalent of a DALY, learning adjusted years of schooling. So just like a DALY, it’s based on kind of the highest quality. So a DALY is based on …

Robert Wiblin: A perfect year of education.

Rachel Glennerster: So a LAY is a perfect year of, the best possible year of education you could have. 460 per $100 spent. You can get 460 Singaporean equivalent years of education.

Robert Wiblin: What’s the program? It’s like, magic pills. I don’t understand.

Rachel Glennerster: Well, so some of these things are … So there are two that come out as sort of spectacularly effective. One is just rearranging kids in a class. So that’s why. So you got two classes …

Robert Wiblin: So it cost basically nothing.

Rachel Glennerster: It cost basically nothing. You have to test the kids so that you put the kids who are, at grade two in the grade two class, and the kids who are at grade four in the grade four class, even though they’re the same age, and they learn much better. So that’s why it’s so phenomenally cost effective because, it really doesn’t cost anything. And then the other one is about information. So sort of sending information over the phone. So these really small nudges. Now none of them transform any kid’s life, but they are so cheap that you get these fantastic returns, and we do very little of that.

Robert Wiblin: Interesting. I guess whatever you have a program that has like close to zero or negative cost, then the benefit to cost ratio becomes a bit meaningless, because it’s not considering a real cost which is like a tension and like figuring out how to do the program, and I guess political capital that you might spend to making it happen. There’s a lot of issues that raise …

Rachel Glennerster: But a good program in education is like one LAY per $100. There were a bunch that did 10-

Rachel Glennerster: … there were a bunch who hit 30. Like they were orders of magnitude difference between the best and the good.

Robert Wiblin: So there’s multiple different things here. Well, it sounded a bit like you were saying that most of the return on the original testing of programs that came from just the handful of [inaudible 01:27:02] most cost effective once you scale them up. So you could also ask like at the point that you, if let’s say you were considering to scaling up all of them individually, like how wide is the range there, but it sounds like the variants will be pretty wild anyway, like really very high.

Robert Wiblin: But is this true like if you test things multiple times so you’re not kind of getting the [inaudible 01:27:18] like you’re not getting regression to the mean on like the highest accidental measurement, and also if you consider that like it might work very well in some places and then like only moderately well in other places. So like all of these things tend to kind of push them, punch them back towards like all having a somewhat more similar impact. Even then you think it’s like extremely high variants across different programs?

Rachel Glennerster: Yes.

Robert Wiblin: Perhaps also just this fact that often you can get like, these little programs that cost very little, that have amazing benefit to cost ratio is but like can’t really be scaled up that much, or you can’t spend that much money on them potentially.

Rachel Glennerster: Well we can’t spend that much money on them, but in terms of like world GDP, but in terms certainly as effective altruists we could spend all of our money in getting those education information ones out. Or, potentially these mobile phone agriculture things, I mean they’re just so cheap. So it’s true that you’re not probably going to end up spending all of your agricultural budget on mobile extinction, but there’s huge returns to getting that right, testing to get it so that it is right and designed in a really good way, in similarly on these education and information interventions, and we could spend money on this.

Rachel Glennerster: And it’s slightly, I get slightly frustrated. I had a conversation with, maybe I shouldn’t use a name, but billionaire. It’s a couple of billionaires who, when we were talking about deworming and they’re like, you mean we could just deworm the whole world for, what was small change for them. And we’re like, yes. And they like, I know.

Robert Wiblin: That’s so disappointing.

Rachel Glennerster: That’s too small. I was like, but you could have done, that the fact that it was that, that was a small amount of money for them, doesn’t mean that they shouldn’t just have done it.

Robert Wiblin: I guess, I just say, there’s like, if you just take like the benefit to cost ratio across like all the individual studies, and it was like very widely dispersed, and then there’s like various things that kind of chop that way a little bit. And the question is like after all these adjustments, like how high is the variance? But it sounds like it is still very high, even if you’ve made some adjustments towards this.

Rachel Glennerster: So I have looked at this a bit in education, and people like David Evans have done things saying, well look, the costs are really different in different environments. But the order actually the rough range of these are really good. These are kind of good and these don’t work, is pretty consistent just because the orders of magnitude is so big. Now that’s not true of all things. But again, these information ones consistently come up as very high.

Robert Wiblin: Because they’re so cheap?

Rachel Glennerster: Because they’re so cheap. If they work and they’re so cheap. And then so I divide things up into, this is something which consistently works, in very …

Rachel Glennerster: … is something which consistently works in very different environments. The information education stuff, you know, US, Chile, Madagascar, we’re getting really different contexts, implemented really different ways, still getting phenomenal returns. Then, there’s the ones that are hard to implement, and they work sometimes, and don’t work others, and that’s … So, again, in education, because I know it better, paying teachers for results really depends on how well you do it. You can really screw that up, and get adverse, unintended consequences. Then, there’s a bunch of stuff that just hasn’t worked anywhere, ever. We really need to stop doing that.

Charter cities

Robert Wiblin: Yeah. Let’s push on from that. You’re not that keen on charter cities, it sounded like. You said that you think only Paul Romer is keen on that. Well, I’m kind of keen on charter cities, at least in principle, so I’m curious to know why I’m probably overrating them?

Rachel Glennerster: I guess, I should say, I haven’t spent that much time thinking about it, because I just thought that the political economy problems …

Robert Wiblin: Okay, right, right, yeah-

Rachel Glennerster: … were just so difficult that-

Robert Wiblin: It’s not going to happen.

Rachel Glennerster: It’s not going to happen, so I’m not going to spend my time worrying about it.

Robert Wiblin: Yeah, okay. That potentially makes sense.

Rachel Glennerster: There were a lot of really high-return things, a difficult political economy, and I’m going to spend my time on that, like, getting people to charge for carbon.

Robert Wiblin: Okay, yeah, okay. So, it’s basically, you’ve got this trade-off sometimes, between how easy it is to get something up politically and then how useful it is. And you just think, given how … Maybe it would be a good idea, but given just the intense political difficulty of making it happen, it’s not worth it, to focus on this one?

Rachel Glennerster: Yeah.

Robert Wiblin: Basically, yeah. Did you feel the same way about special economic zones, and things like that?

Rachel Glennerster: Yeah, so that’s what I was going to say. Right, to some extent, what the argument there is about is you could potentially do half the things he’s talking about, in a special economic zone, so why not do that? But I don’t know, again. I don’t know the literature enough on the special economic zones. It’s a quite hard thing to test. Look, we absolutely know that institutions matter. There’s extremely good evidence on that. We also know that they’re extremely hard to change. Right, those are the two things we know.

Rachel Glennerster: That doesn’t mean that we shouldn’t keep working on ways to practically change them, and I’m all in favor of that, and I’ve done some work on that. And, whether it’s improving democracy or decentralization, or … there’s all sorts of different things that you can do to try and make institutions more responsive to the needs of the population. And, we know that rule of law is really important, because we see that, hundreds of years later, the places that had bad rule of law are still worse off. So, we know that that matters.

Rachel Glennerster: Given the persistence, it’s not clear to me that just putting place a new framework actually delivers those results. So, the best evidence on those kind of rule of law and institutions work comes from countries where there was different institutions, sort of 100, 200, 300 years ago. They are now under exactly the same rule of law, but the places that were under the worst rule of law 300 years ago are now … are still significantly worse off. That suggests to me that just laying a new thing on top is not actually going to solve the problem.

Robert Wiblin: I guess there is this interesting thing with … if something’s very persistent, basically like, this year determines next year, then it’s like, the impact of trying to change it, it kind of cancels out the fact that it’s very hard to change, but, if you did, it would last for ages, if you’re taking a long-term view. But it does suggest, yeah, that you’re going to face significant head winds, trying to make things any different, although, if you do succeed, then that’ll be great.

Rachel Glennerster: I think the right thing that we should be thinking about, and maybe charter cities is an example of that, but we should be thinking about practical things that we could change, and testing them out, whether they have any impact. It’s interesting, because some things, like craters for women in India, you would think, really long-term discrimination against women. And yet, a specific supreme court ruling, that led to women having more political power, real political power, over real money, in India, had very quick turnaround impacts, on real lives. Not just perceptions of women, but actually improvements in resources, and outcomes within those communities. So, yes, it’s hard to change things, but, if we keep testing, there are examples where we have managed to change things, so let’s keep looking for those.

Gender equality interventions

Robert Wiblin: Yeah. And are there any other really high-value interventions in gender equality, that you would like to highlight?

Rachel Glennerster: Increasingly, I am getting convinced that these safe spaces for adolescent girls, we now have a number of those, with positive effects. We still, again this is an example where there’s one not-good one, where implementation was poor-

Robert Wiblin: Is this at school, sorry, universities or-

Rachel Glennerster: No, no, no, this is in communities. So, I evaluated one in Bangladesh. BRAC does a lot of these. They had an evaluation in Uganda that was very successful. There’s just been one coming out in Sierra Leone, where they seemed to protect girls from pregnancy during the Ebola crisis. There was a massive increase in adolescent pregnancy during the Ebola crisis. So, that’s something which I think is something, definitely, we should look at. I think, the standard girls’ education, I was always very skeptical, because that was based on pretty much no evidence, but now we’re building evidence, and it actually turns out it was quite a good thing.

Rachel Glennerster: Family planning, there’s quite a bit of evidence from different contexts, that allowing access to family planning allows women to … Because they have more control in the future, they actually then invest in their education now, and they take on different roles and are more likely to work, and it’s sort of, that forward-looking, if you reduce the costs … If you don’t have family planning, you know you’re going to be having kids forever, and there’s no point in going to school. There’s no point investing in other human capital. So, I think that’s pretty general, too.

India, carbon and pollution

Robert Wiblin: Yeah. If you could advise Narendra Modi, the PM of India, on one policy issue, hopefully get him to take you quite seriously, what do you think you would talk about, in that meeting?

Rachel Glennerster: I would try and persuade him … So, this is assuming that he would listen.

Robert Wiblin: Yeah, setting aside political reality, maybe-

Rachel Glennerster: Yes, okay.

Robert Wiblin: … is the spirit of the question.

Rachel Glennerster: I would try and persuade him to put in place markets for carbon.

Robert Wiblin: Oh, interesting, really? Okay. Explain that, sorry. I didn’t expect that.

Rachel Glennerster: Right. For a couple of reasons. So, one is just climate change is going to have huge impacts on the poor, and India is a big emitter of carbon. I firmly believe that, if you get prices right, there’s lots of things that people would do differently, if the price … That are reasonably cheap, but we’ve so screwed up prices that they don’t have the incentive to do it. So-

Robert Wiblin: It’s music to my ears, as an economist, but carry on.

Rachel Glennerster: So, I think that’s one, if you’re looking for one thing, that would then set the right incentives throughout the system. And the final point is the health impacts in India of burning coal are just extraordinary, unbelievable health costs of all those coal fire power-.

Robert Wiblin: Yeah, I was listening to a radio program the other day that was saying that it was taking about 10 years off the average life, for people in some of these cities like Delhi or Mumbai. I was like, it’s like the equivalent of smoking cigarettes, maybe even more so, which is absolutely crazy.

Rachel Glennerster: Yes, it’s the equivalent of heavy smoking every day. And you think about kids, breathing that in.

Robert Wiblin: Yeah, the air pollution thing makes sort of sense, but wouldn’t you then perhaps want to put in a program that just taxes air pollution? Or, do you think that taxing coal comes pretty close to doing that, or taxing carbon in general, is pretty close to an air pollution tax?

Rachel Glennerster: I would also love to do stuff on pollution, but a lot of this is coming from coal, and obviously then you also have climate impacts. I’d have to work through … I haven’t gone through all the detailed numbers of how much of those particulates are coming from coal, and how much are coming from other things. But the double whammy of-

Robert Wiblin: Both climate change and saving just very large numbers of lives.

Rachel Glennerster: Yes.

Robert Wiblin: Yeah, on this program now, talking about people who are trying to create city-wide air purification, so that you’d build buildings that would suck up air from the surrounding city and then purify it, basically like the same thing that you’d have in your house, but like a building that does that-

Rachel Glennerster: Wow.

Robert Wiblin: Yeah. It sounded like it potentially might work, but it’s perhaps too expensive for India to just get it. There’s-

Rachel Glennerster: Well, there’s a relatively easier solution-

Robert Wiblin: Just stick a filter on the plant itself.

Rachel Glennerster: The crazy thing is that it’s not that much more expensive to build a solar power station now, versus a coal power, and, if you added in the health costs, let alone the environmental costs, it’s so much cheaper.

Illicit financial flows

Robert Wiblin: One thing you spoke very positively about was helping to stop illicit financial flows. Is that something that listeners can potentially help with, in their career, or is that something that it’s really a matter for governments in the developing world to work on? Or, locals? I suppose, potentially, we could try to reform policy in the US and UK and Australia, right, inasmuch as we’re helping with some of these illicit financial flows, which I’ve heard we do.

Rachel Glennerster: Yeah. I think a lot of these illicit financial flows flow through us. In general, people in the West are better at influencing policy … Well, I don’t know, but in some ways it can be easier to influence policy in your home country. You have to spend a lot of time understanding the political economy if you try and influence policy elsewhere. So, I think that’s absolutely something that we should be thinking about. Again, I haven’t done a full cost-benefit analysis, but if you think about the money that’s coming out, you think about how it’s distorting policy decisions-

Robert Wiblin: Interesting.

Rachel Glennerster: … because-

Robert Wiblin: In the developing world?

Rachel Glennerster: Yes.

Robert Wiblin: Okay, yeah. So programs are being designed, such that, like, money can be siphoned off. Is that what you mean? Or, like-

Rachel Glennerster: Yeah.

Robert Wiblin: … decisions are made based on what will facilitate friction?

Rachel Glennerster: So, for example, Mozambique is now in a massive debt problem, because it took on a highly corrupt loan, which was laundered in London. If that loan hadn’t happened, Mozambique would have an awful lot more money to spend on poverty relief.

Robert Wiblin: So the money was just stolen outright, more or less? It was like siphoned off in some way, through banks in London?

Rachel Glennerster: There are a lot of deals that are done, and a lot of poor policy decisions are made, including building coal-fired plants instead of solar ones, because someone is given money to do that. Now, stopping the illegal flows out of the country doesn’t … They could still benefit from the corruption internally, but a lot of that money does go outside the country. So, a lot of people worry about it, saying, “Well, that money could be invested in the country.” I don’t think that’s the main benefit.

Rachel Glennerster: I think the main benefit is, if we could get to a point where policies, decisions, in countries were based on what was good for the country. So, you could have more transparency locally, but it’s actually potentially easier to have transparency wider, because a lot of this money then does go into bank accounts in the West. And, if people knew that there was a risk that they wouldn’t be able to keep that money, because someone would confiscate it, that would reduce the likelihood that they would take that-

Rachel Glennerster: Yeah, it would change incentives, exactly.

When countries are open to major economic reform

Robert Wiblin: The other day, you talked pretty positively about the potentially massive returns you get when a country is open to major economic reform, and you go out there, and say, “This is how you ought to structure markets, based on what we know.” How crazy would it be to just design DIFD to just wait around for whatever countries are open to massive economic reform at that point, and then just flying in tons of people to try to give the best possible advice on that question? If that’s the program that has the biggest benefit to cost ratio, but, at any point in time, there’s only like two countries in the world that are receptive to that kind of advice, maybe you should just be like waiting around, and then flying to those two countries, like between them?

Rachel Glennerster: Well, that kind of assumes that you can fly in and give advice without knowing the country, and also that pouring in the money doesn’t cause problems in some way. But I-

Robert Wiblin: I recognize there’s some downsides to this proposal, yeah.

Rachel Glennerster: But, I absolutely am advocating that we should be more flexible, and certainly put people … For example, I mentioned Ethiopia as somewhere where there are massive changes happening at the moment, and I think we ought to absolutely be doing what we can. Now, there’s a question of, is it DFID, is it the World Bank, do we give the right advice? Because, sometimes what happens is, you get to the end of the world …

Robert Wiblin: Hopefully not.

Rachel Glennerster: End of a war, and we fly in lots of experts, who don’t know anything about the country, and say, “Okay, here’s your forest regulation,” was the example being given earlier today. You know, “Your forest regulation should be this.” And we give them the UK’s forest regulation, which they’re not going to-

Robert Wiblin: May not have good external validity.

Rachel Glennerster: Exactly. You’ve got to do it smart. But, if you think about what we did … I’m old enough to have been around when we were giving advice, post the end of the Cold War.

Robert Wiblin: Seems like we screwed the pooch on that one, right? Or is that not the case?

Rachel Glennerster: No, look at … There are a lot of countries that were in really dire straits, and moved to market economies.

Robert Wiblin: Okay, seems like in Eastern Europe, and do you mean like Poland, perhaps, or-

Rachel Glennerster: Yeah.

Robert Wiblin: Interesting.

Rachel Glennerster: Now, Russia’s maybe not been such a success.

Robert Wiblin: No. I guess, people blame us, I think, for the oligarch problem, that all of these public assets were sold off for next to nothing, and then a bunch of people became really rich, and now they control the politics, and so there were some problems, even if there were some wins as well.

Rachel Glennerster: Yes, exactly. I think, some countries, this was done better than others.

Careers

Robert Wiblin: All right. Let’s push on to some career-related questions, like advice that potentially listeners can take. So, you’ve been at DFID for about nine months now? Nine, ten months. Yeah, do you have any kind of stories where you think your presence has made a difference already, that you can potentially talk about publicly? Or is it maybe too soon for that, or it’s too sensitive?

Rachel Glennerster: The first piece of advice on policy influence is, never claim credit. Always make someone think that they did it.

Robert Wiblin: Right, well, we’ll bleep out all the specifics.

Rachel Glennerster: No, I was thinking about what would be … What I can say is, where I’ve given some advice. And, I’m not at all saying that it was because of me that this happened, but it’s an example of where having done analysis can help inform a decision, right? I think that’s the right way to think about it. I had spent a lot of time working on Sierra Leone in the past. I know their data quite well, so it was one of the DFID countries that I went to visit. It wasn’t so much actually influencing what DFID was doing, but I was talking to the World Bank and the IMF, about their programs, and one of the conditions that the World Bank had had … There was a big need to increase revenue, and they had been asking for a long time for the government to increase the revenue from fossil fuels, so, from petrol. They massively subsidize petrol. So, really bad for the environment, really bad distributional impacts, really bad for revenues. They were spending loads of money on it.

Rachel Glennerster: So, that was fine, but then, the second thing they were talking about, and was originally a bank condition, was to increase the excise tax on rice imports. And I was able to say, or I argued with them, saying, “Look, you’re increasing fuel prices. Fuel price is actually a big part of the cost of food in rural areas, because you ship it in the hungry season. You are about to enter the hungry season.” And I showed data about what happens to the price of food, what happens … 90%-plus of people are hungry in Sierra Leone, skip meals in Sierra Leone in August. You do not want to be raising the price of rice in August. And, actually, it’s imported rice that people eat in August. So, that didn’t happen.

Rachel Glennerster: I don’t know whether it was because of the evidence that I showed, but that seems like a pretty big policy … It seems like it was open. You could bring some data to bear on it. Even if you’re going to do it, don’t do it, two whammies at a time, when people are literally starving. That’s the kind of thing where I think, it’s not an RCT, it’s just basic data of looking at seasonality. Also, the argument was, well, people in the rural areas don’t eat imported rice. Well, actually in August they do. So, just having these basic descriptive data was actually, I think, quite important.

Robert Wiblin: There’s two projects in the effective altruism community, Charity Science Health and Charity Entrepreneurship, that have actually started programs in the developing world, that focused on micronutrient supplementation, I think, and also vaccination reminders. I’ve been trying to draw out if there’s any interventions that you think are particularly underrated, that could be really useful for people to work on scaling up, or delivering, or testing. Are there any others that I haven’t come up, that you’d like to flag as potentially good options for listeners to try to deliver themselves, or to talk to other people about?

Rachel Glennerster: That’s a good question. I think there are not enough organizations that take one thing, and try and push that one thing at scale, really iterate, and try and do it at scale. Some of this education information stuff might be an example of that. I know effective altruists sometimes have questions about education. Is there strong enough evidence that education’s a good thing? If you care about education ,I think that’s a big thing to do. There are people working on that.

Rachel Glennerster: I have to think a little bit more about … I don’t have, off the top of my head, a list of things that I think we ought to be doing more of. It’s interesting, though, because it is the case that DFID can’t do most of these things, and sometimes, when we say, “Well, look, there’s evidence that this is a really good thing, can we commission someone to do it?” And there aren’t organizations there. But they need to be at scale, and that’s the thing that I worry about, right?

Rachel Glennerster: It’s no good being a small implementer doing this. Certainly in terms of us being able to commission. We commission sort of 50 million … I was looking at recently, a proposal for 170 million on education in Nigeria. So, that’s a bit of the hurdle of effective altruists starting their own organizations to do this, versus joining another organization. The trade-off is, some of those other organizations don’t want to do one thing. They want to do-

Robert Wiblin: All kinds of different projects at once.

Rachel Glennerster: … all kinds of different things. And I don’t mind people doing all kinds of different projects, like, Evidence Action does a number of different … But they’ll only do one at a time. They take a lot of different evidence-based things and turn them into scale programs. But they don’t think that, in order to do deworming, you also have to do toilets and training and shoes and … They just do the deworming, because that’s the thing that the evidence is.

Robert Wiblin: It’s just really weird that there’s this culture of yet wanting to have programs that are holistic, that do everything. Because you don’t, for example, find that, like, Starbucks wants to also scale up and run hospitals, or run all kinds of different products at the same time. Typically, a business tries to nail one product really well at a time. But that doesn’t seem to exist as much.

Rachel Glennerster: I’ve been thinking a lot about this, since I’ve been at DFID. I think there’s some very specific incentives, within the system. Which is, if you want to be able to show people your project working, if you’ve, say, done a really cheap thing that makes millions of people marginally better off, you’ve got nothing to show, nothing to take pictures of, nothing to take a minister to.

Robert Wiblin: Right. So it’s like a portfolio of thing that look good and things that are useful? More on the looking good, and more on the useful side.

Rachel Glennerster: But it’s also partly, I think, in our individual psychology. If we’re supporting a school, we want to go to a school where the kids are looking happy and they’re well-fed, and there’s a roof on the classroom, as opposed to, well, they got a text reminder, and they’re 10% more likely to be in school. That doesn’t give you … Maybe that doesn’t- [crosstalk 01:51:30]

Robert Wiblin: Well, it does to me.

Rachel Glennerster: It does to you.

Robert Wiblin: Warms my heart.

Rachel Glennerster: Yes, but, unfortunately, there’s a psychological thing, which is, there are a lot of people who don’t think in numbers, and it doesn’t work for them. I’ve been working on some political stuff, and the implementers we work with just really want to do it beautifully in three villages, and we’re trying to say, “No-

Robert Wiblin: No.

Rachel Glennerster: … whole country, whole country. How do you do this for the whole country? And there’s something in the development community … There’s a number of different factors. There’s something going on which I haven’t quite understood, but I have some insights into, as to why people want it to be perfect in a few places, rather than kind of okay, or slightly better, everywhere. There’s also a fundamental belief in the development community that fixing one hurdle doesn’t work, that people need all of the hurdles removed.

Robert Wiblin: This is the Millennium Villages, kind of, like, yeah?

Rachel Glennerster: Yeah, and sometimes that’s true. Sometimes you get complementarity, but it’s not always the case.

Robert Wiblin: And you also get lots of specialization.

Rachel Glennerster: I think part of the problem has been evaluations that go in and say, “Well, this program didn’t work.” Evaluations being, not RCTs, but the more qualitative evaluation, which goes in and says, “Well, this program didn’t seem to be working. Why is that? Well, that’s because there was this other barrier.” And maybe it’s just, it didn’t work. Maybe just, those kinds of things, just don’t work. But then, if you read all those evaluation reports, you would say, “We have to do all of these things together.”

Robert Wiblin: Last few questions. What were the best decisions that you’ve made in your career, in retrospect?

Rachel Glennerster: I guess, I moved out of domestic policy and into development, and I think that was a good move. I’m not saying it’s right for everyone, but I think the amount of impact that you can have in development is really big, and I put the investment into getting more economics, and more ability to do data analysis. I think that paid off. There was a time when I thought, “Oh, I can muddle by with what I’ve got.” But I ended up putting considerably more investment in learning the tools, and while I was learning them, part of me was thinking, “I’m never going to use this.” But just mastering it meant that I just went up a level in my ability to think through the ideas, and I think that was a good investment for me.

Robert Wiblin: Yeah, that’s one I hear fairly often. I know there’s quite a lot of listeners who are early on in their career in the Civil Service in various countries. Do you have any general advice for them, on how potentially they can go further or have more impact?

Rachel Glennerster: Yeah, I am a big fan of the Civil Service. That’s where I started life. I would have stayed forever if I hadn’t met Michael, and … He persuaded me to move to the States. You can end up having a lot of influence, at least in the British style, you know, Britain, India, Pakistan, the places that I know. The US, it can sometimes be harder, because a lot of the top jobs go to political appointees. What I would say though, is that sometimes it’s hard to keep up on the literature, especially as an economist, and I worry that people read the things that are easier to read, and the things … They Google what’s … And, the think tanks are very good at putting out reports that seem-

Robert Wiblin: Look splashy and …

Rachel Glennerster: Yeah, they look splashy. They also are designed to look like they are answering the question that you want answered, but they’re doing it badly. Because they’re constantly looking at, what the Civil Service, what the government, what are the keywords? And they’re doing reports on that. So, coming back into the Civil Service, I’m trying to push people to say, “No, don’t read the think-tank report, let’s stay up on like, the-

Robert Wiblin: Read the paper.

Rachel Glennerster: Read the paper, right. Because, often, the think-tank report is … Now, look, I’m not trying to dismiss all think tanks. I’m using that as a generalisation. Don’t read the thing that is a cross-country regression, or the back-of-the-envelope estimate in a glossy report. Read the serious analysis. Now, maybe it’s an overview of the serious research, but try and keep up on that. I have colleagues in my unit who are reading the top economics journal. I don’t know where they find the time, but somehow they do, and they’re up-to-date on those things. I think that is just really hard to do, but really important, in terms of, if you’re thinking about this, 80,000 Hours question of, my marginal benefit, versus someone else who could be in the job instead of me. Make sure you’re that marginally better person, by making sure that you stay up-to-date on what is the best evidence.

Robert Wiblin: Yeah. In the other interview, you mentioned that you thought starting a successful business in the developing world was potentially underrated. Do you know of any instances of people from the US or the UK or Australia actually going and managing, to successfully do this in a developing country, who people could learn from? Just the names of the businesses would be plenty, I think, to go and study them.

Rachel Glennerster: I know people who’ve started businesses. It’s a bit early on to say whether they’ve really changed things. This is partly coming from the literature, showing that there’s huge potential improvements in productivity in most companies in the developing world. Some simple, basic improvements in management practices could massively improve things. Now, maybe I’m wrong, that someone coming with a lot of analytical background would be able to help push those management practices through, but, certainly the evidence is there that there’s a lot of scope to-

Robert Wiblin: There’s a lot of low-hanging fruit.

Rachel Glennerster: There’s a lot of low-hanging fruit in businesses, and we know that the poor want jobs. They don’t want to run their own firms, they want a job in somebody else’s firm. So, we know that, and also, just what I’ve been able to see when we work with partners, some of the basic things that we can do to improve their MI systems. Basically, we do this in order to get the ability to do research. We researchers do this in return for them agreeing to randomize. And, I can tell you, we do massive improvements on people’s data systems, on their basic decision-making, once we’re in partnership. So, that’s where my idea came from. If you were inside those organizations, you could potentially have a big impact. And, you came with good analytical skills, and good coding, and ability to build a better MI system, I think there’s huge potential there, if you found the right group who wanted to take you on.

Robert Wiblin: Yeah, I think I know some of the reports that you’re referring to there. They’re quite eye-opening, so I’ll try to find them and stick up links for listeners to take a look at. So, just one last question. It’s pretty often, like, civil servants can find it’s hard to lobby for something internally, without an external push to interest politicians, or perhaps political appointees within the Civil Service. Is there anything that listeners can push for that would potentially help you, in your job at DFID?

Rachel Glennerster: Okay, so you’ve just said that I’m not allowed to say that.

Robert Wiblin: Oh, interesting.

Rachel Glennerster: I think the answer has to be-

Robert Wiblin: Well, you might be able to say that, “Here’s something that I support, that other people are not necessary that interested in within the institution, or …”

Rachel Glennerster: Well, just in general, there is an awful lot of bad rhetoric around aid that is just misinformed, and, get out there and be part of that debate, I think. In particular, telling the story of the improvements that we have seen. Like, the majority of the populations in the countries that we live in, think that the poor are getting poorer, and they don’t know-

Robert Wiblin: And that aid doesn’t work.

Rachel Glennerster: And that aid doesn’t work. And I’m not saying that all aid works, or whatever, but they’re saying, “We put all this money in, and people are still poor.” Like-

Robert Wiblin: We didn’t put that much money in, and things have gotten a lot better.

Rachel Glennerster: Exactly. So, go out and tell some successes, I think is pretty important, actually.

Robert Wiblin: Fantastic. Well, my guest today has been Rachel Glennerster. I’m so glad we got time for a second session here. Thanks for coming on the show.

Rachel Glennerster: Thank you.

—

Thoughts from Eva Vivalt: I think Rachel and I are mostly on the same side. I would agree that results from different studies tend to be very heterogeneous, and my paper “How Much Can We Generalize from Impact Evaluations?” highlights this. Unfortunately, many organizations continue to act as though there is one true effect of a given intervention on a given outcome that holds everywhere, and I think my paper should be read as in part a reaction to this. In terms of modelling the effects, I agree that careful models could help, though practically speaking it’s going to be hard to build good models when you have got very few studies to draw upon. That’s where you may need to rely more on priors — and it was great to see Rachel referencing Stefano DellaVigna’s work with Devin Pope on this, because I’m currently collaborating with Stefano on building a platform to help gather ex ante forecasts of social science results more systematically. We are trying to solve the coordination problem that we see brewing, whereby if a lot of researchers start independently soliciting priors it could become a huge burden on expert forecasters. Another advantage of the platform is that it provides third-party, credible timestamps of when predictions were gathered and transmitted back to researchers. I think there is a lot we still have to learn about when we can trust our ex ante predictions, but I’m very excited about the potential for this line of work to shed some light on that question.

Robert Wiblin: Just a reminder about the Future Perfect and Future of Life podcasts, which you can subscribe to if this show isn’t enough for you.

As always we’ve linked to a range of papers and reports that Rachel referred to throughout the episode, in the blog post associated with the show. I’d strongly encourage you to sometimes check out the resources we put together for people who want to learn more. We link to those blog posts in the show notes which you can access from your podcasting app.

The 80,000 Hours Podcast is produced by Keiran Harris.

Thanks for joining – talk to you in a week or two.