Transcript

Rob’s intro [00:00:00]

Rob Wiblin: Hi listeners, this is The 80,000 Hours Podcast, where we have unusually in-depth conversations about the world’s most pressing problems, what you can do to solve them, and why my days of chasing sugar daddies are over… or are they? I’m Rob Wiblin, Head of Research at 80,000 Hours.

When I asked my Twitter followers who we should interview about global health and development, Karen Levy was the most requested guest, for reasons that should be obvious in just a minute. Caitlin Tulloch replied that, quote, I “must talk to at @KareninKenya, she’s the best!” — and I’m glad she made that suggestion.

Just before that, I want to let you know that 80,000 Hours itself is currently hiring for an operations specialist based in London.

Our mission is to get talented people working on the world’s most pressing problems. We expect to increase our staff count by about 50% in 2022, and to continue scaling from there. We’re looking for an operations specialist to help build the organisation and systems that will support this growth.

They’ll run the team that manages our office, help scale our internal processes as we grow, and oversee a range of other projects according to their personal fit.

The starting salary for someone with one year of relevant experience would be about £58k, more for candidates further into their careers.

Applications for that role close Sunday, April 3.

You can find the full job description and apply, either on our job board at 80000hours.org/jobs or our blog at 80000hours.org/latest.

OK, without further ado, here’s Karen Levy.

The interview begins [00:01:30]

Rob Wiblin: Today, I’m speaking with Karen Levy. Karen is a veteran of the global health and development scene, having held senior leadership roles at several organizations at the heart of the randomized evaluation revolution. She established Innovations for Poverty Action’s country office in Kenya in 2006 in collaboration with recent Nobel Prize–winning economist Michael Kremer, and served as the Country Director there for five years, during which the Kenya team grew from about 15 to over 200 personnel.

Rob Wiblin: During that era, Innovations for Poverty Action incubated a project called Deworm the World, which was founded in response to a randomized trial indicating that deworming had a major impact on kids’ school performance and later labor market outcomes. Deworm the World aimed to massively scale the delivery of deworming tablets by working with governments to do school-based deworming, starting in Kenya. As part of that, Karen led a team providing technical and logistical support to senior health and education officials. Today Kenya’s school deworming program reaches over six million children a year, and has inspired similar programs in many other countries.

Rob Wiblin: In 2014, Karen cofounded a new umbrella organization called Evidence Action, where she started the Evidence Action Beta team, which aimed to test and build a viable path to scale for promising evidence-based interventions. There she worked on a new development program called No Lean Season, which was supported by GiveWell and went through Y Combinator’s nonprofit incubation process. After some further beta testing, No Lean Season was closed down in 2019 — becoming one of very few development projects to voluntarily shut its doors.

Rob Wiblin: Since then, she has worked at her own strategic advisory firm for global health and development work, known as Fit for Purpose. Karen did her PhD in development planning at the University of London; has lived in Nairobi, Kenya for over 20 years; speaks fluent Swahili; and tweets from her account @KareninKenya. Thanks for coming on the podcast, Karen.

Karen Levy: Thanks so much for inviting me.

Rob Wiblin: I hope we’ll get to talk about lessons you learned being involved in scaling deworming as well as testing a seasonal migration program. But first, as always: what are you working on at the moment, and why do you think it’s important?

Karen Levy: These days I am working on a new organization called Fit for Purpose, which I started with my amazing cofounder Patricia Mugambi-Ndegwa. We help leaders in global health and development and philanthropy who care about evidence and data-driven decision-making to scale the impact of their work. We’re particularly focused on work at the intersection of evidence, policy, and scale; and on helping evidence-based development partners work more effectively with governments and other policymakers. And we’re really excited about this as a means towards really helping to turn up the volume substantially on the impact that we can have in lower-middle-income countries.

Rob Wiblin: Nice. Yeah, we’ll come back to Fit for Purpose and some of that consulting work later in the conversation.

Funding for effective altruist–mentality development projects [00:04:07]

Rob Wiblin: Let’s talk now about some trends you mentioned that you are excited to be seeing at the moment. First off, how has the availability of funding for effective altruist–mentality development projects been shifting in recent years?

Karen Levy: I think that there has never been a more exciting time to be working in the field of poverty alleviation and global health and development. And that is because we are finally starting to see a shift in which the market for good solutions and good programs is aligning. At the end of the day, organizations need to fund the work that they do, and so they have to respond to the preferences and the needs and the desires of funders. So the more that funders are demanding evidence-based, impact-oriented programs, the more that people who care about doing that kind of work are able to do so.

Karen Levy: Looking at the landscape now, we have an incredible and growing infrastructure of organizations that can generate evidence — and that are more and more sophisticated about doing so alongside the people who will be using that evidence. We’ve got new organizations that are good at delivering impact — either themselves or in partnership with governments. And now more than ever, we have a real increase in the resources that can unleash all of these trends. So to me, this is what it should look like. There should be more resources available to do good interventions than there are good interventions, because that’s what’s going to make people make more of them. And so I just think that it’s a very, very exciting time to be doing the work we do.

Rob Wiblin: The big funder that I’m familiar with is GiveWell, which is working with Good Ventures and giving something like half a billion dollars a year, I think. Are there other major funders? Or maybe just the mentality has spread to other foundations and governments, and so it’s more just all over the place these days?

Karen Levy: Certainly GiveWell inspired donors, and Good Ventures and Open Philanthropy are really leading this and are major players in this. And recent announcements that they’ve made — about increasing funding over the next few years and beyond — I think has gone a long way to changing the landscape.

Karen Levy: But it’s not only them — I think there are other funders in this space. We, for a number of years now, have funders following this find-test-scale model: Development Innovation Ventures at USAID, Global Innovation Fund, a new vehicle in France, FID, that is based on that same model. We also have some new and exciting examples in ‘big-bet philanthropy’: Co-Impact, The Audacious Project. These are people in philanthropy who are thinking really boldly about swinging big and really solving big problems — and doing so in much more risk-seeking ways, in a good way.

Pre-policy plans [00:07:45]

Rob Wiblin: I know one methodological innovation that you’re excited about at the moment are ‘pre-policy plans,’ which I actually hadn’t heard of until you mentioned them. Could you explain for the audience what they are?

Karen Levy: The idea of a pre-policy plan is inspired by a pre-analysis plan — which we know is common, but becoming even more and more typical in the social sciences. And this idea came up when I was doing some work with an organization in Botswana called Young 1ove. Now, Young 1ove is led by Noam Angrist and Moitshepi Matsheng, an incredible team of people that they lead in Botswana. Several years ago, they were replicating a study that had been done in Kenya many years earlier by Pascaline Dupas that provided relative risk information to adolescents in order to help them make better decisions about their sexual partners and reduce their vulnerability to HIV infection.

Karen Levy: So this intervention had been tested in Kenya, and had some really impressive impacts on pregnancy — which was used as a proxy for unprotected sex at that time — but then nothing had been done with it after that. And so Noam and Moitshepi and their team had taken those results and adapted them to the local context in Botswana: refreshed the intervention, built a new curriculum around it. But being an organization that cares about evidence, they decided to subject it to a new RCT in the Botswanan context before scaling it up.

Rob Wiblin: Yeah. This is the No Sugar intervention, right?

Karen Levy: Exactly. This is No Sugar. It’s called No Sugar.

Rob Wiblin: In reference to sugar daddies.

Karen Levy: Sugar daddies. That’s right. And the idea being that, traditionally the options given to adolescent girls are: have sex and die, or be abstinent. I mean, these are not realistic choices for people. And I think the real brilliance behind Pascaline’s study was empowering girls with information that allowed them to manage risk: explaining to them that older men are riskier and more likely to be HIV positive, and so by choosing to have relationships with younger partners, they could reduce their risk of HIV. And so that was the intervention.

Rob Wiblin: And they decided to test it again in this new context and this new era. And it’s just as well that they did. Do you want to explain why?

Karen Levy: Exactly. Now of course, while the test was happening, we had no idea if it was going to work or not. And the Young 1ove team had done a really extraordinary job in developing relationships with government partners. They were doing this program in schools. They had gotten the Ministry of Education all excited about it. And ironically, you often worry that if you find evidence that something works, no one’s going to take it up and run with it. But there’s also the converse problem: if you find something doesn’t work, you don’t want people to scale it up, right?

Karen Levy: So in discussing this dynamic, before the results of the study came out, we decided to work with the government partners to put together what we called a ‘pre-policy plan.’ The idea was to game out in advance, “What will we do if the results are X? What will we do if the results are Y?”

Karen Levy: And the reason that we did that is, when you’re thinking about a hypothetical future, it’s very easy to assume that you’re going to be rational — that you’re going to follow the evidence. But the reality is, when you’re standing there later and you’ve got some data that it didn’t work, there is this enormous temptation to say, “Oh, but you know, it would have worked if it had just been different,” or vice versa.

Karen Levy: So it was creating a space in which there was nothing on the line, and everyone could really think about in theory what they would do. When you create an environment where everyone cares about evidence and we want to be evidence driven, it’s very easy to get people to say, “Of course we won’t scale this up if it doesn’t work, and of course we will scale it up if it does.”

Karen Levy: And then sure enough, when the results came in and they were, very sadly, disappointing, it was much easier to connect people to that mindset that they had been in beforehand. You know, people really liked the program. They wanted it to work. It was an incredibly charismatic group of people doing this. And so everyone’s first reaction was, “Oh, but let’s do it anyway.” But when we looked back at that pre-policy plan, it really helped.

Rob Wiblin: “But we said we wouldn’t.”

Karen Levy: Exactly, exactly. I think the other thing that’s really useful and important about a pre-policy planning process — which I subsequently used in other contexts — is that it’s a wonderful tool for aligning the research and policymaker perspectives. Very often there’s not enough communication upfront, and what this can lead to is evidence not being useful because it’s not available on time, for example, or the results are not applicable to a population that policymakers are actually needing to make policy for.

Karen Levy: And so with a pre-policy plan, the idea is to get people to sit down in advance and really line up: “What are the questions we’re asking? What are the outcomes we’re measuring? When are we going to have this evidence?” alongside “What decisions are being made as policymakers? What choices do they actually have? When and how are resources getting allocated?” So researchers can make their work relevant and responsive to the actual choices policymakers are making and vice versa.

Rob Wiblin: Yeah. So just to explain what happened with the No Sugar intervention. I guess the idea was that these young women are potentially forming partnerships with men who are in their 40s or 50s, and they don’t realize that that’s the group that is most likely to have HIV and so might be the most dangerous to sleep with. The intervention was to inform young women about the facts of who was most likely to have HIV.

Rob Wiblin: When it was tested in this new context, they found that they were successfully convincing women that it was men in their 40s who were most likely to have HIV. But their preferred partners, and the people they were most likely to be forming partnerships with, were people in their 20s. And so in fact, if they were responding to the evidence in the most natural way, they would think, “Well, actually the people I’m sleeping with now are the least dangerous. And actually, maybe I should do it more. I should be even more relaxed about it.”

Karen Levy: That’s right.

Rob Wiblin: The most straightforward, most likely, implication would be that it would have the exact reverse effect of what was desired.

Karen Levy: That’s right. And there was no real way to know that before it was tested. And as a result, Young 1ove and their government partners took the really courageous step of rolling it back, going back to the drawing board, and trying new approaches. They’ve designed new interventions based on that evidence and are currently testing new alternatives to that.

Karen Levy: And they’ve also expanded into other new work. They are working with the Ministry of Education to expand Teaching at the Right Level, which is another evidence-based intervention that’s having a tremendous impact on literacy and numeracy. So yeah, there are very few examples of this, and they’re one of them.

Rob Wiblin: Pre-policy plans are, I guess, figuring out under what conditions would you want to scale a program up, under what conditions would you hold steady, under what conditions would you cancel it — but ahead of time, before you get biased by what the results you’re getting actually are. It seems like in this case, the results were very clear cut and quite devastating. It’s interesting to me that even in this case, people kind of wanted to push on. It’s so hard to let go of a program, even when the most likely outcome seems to be that you’re causing harm. It’s possible that having this conversation ahead of time might have made the difference between continuing the program and not.

Karen Levy: Yeah. I think the truth is that most people want to do something. And I can relate to that: I am action-biased, so all else equal, I’d rather do something than nothing. So I understand where it comes from. And in this case, as you say, I think it was very clear cut. But often it’s not, right? So the other thing that a pre-policy plan can do is for some tough conversations about “How much impact is enough to make this worth doing?” or “What if it’s effective for some subpopulation, but not for another?”

Karen Levy: These are all questions that are much easier to really think through and interrogate when you don’t know the answer. And so, just in the very same way that a pre-analysis plan forces a clarity of thought ex ante, I think there’s a lot more that we can do in the global health and development space to be thinking about how we would use new information before we get it.

Rob Wiblin: Yeah. For those who don’t know in the audience, a pre-analysis plan is figuring out how you’re going to analyze the data before you have the data — so it reduces the degrees of freedom you have in how you analyze it. And so it means it’s harder for you to either deliberately — or more likely just accidentally — put your thumb on the scale and cause yourself to get a particular result because that’s what you want to hear.

Rob Wiblin: Doing that is challenging though, because you’re blind: you don’t have the data yet, and yet you have to anticipate what the data might indicate and then figure out what you would do in each of those different situations. So it’s something that’s become much more popular, but I think it’s most popular within academia, where people have the kind of technical expertise to actually make one.

Rob Wiblin: I imagine it’s also challenging to do a pre-policy plan, because again, you have to consider many different eventualities and how you would interpret those different hypothetical results. I suppose this is maybe something that academics can build up more experience with, because this is really in their wheelhouse, and then they can transfer that information into governments and into NGOs — where perhaps they don’t have quite as much experience with this level of data analysis.

Karen Levy: Well, remember that a lot of what a pre-policy plan is about is what policy choices you have, right? And frankly, those are often more constrained than research choices.

Rob Wiblin: Right.

Karen Levy: Your intellectual freedom to explore data can be unlimited, but your actual real-world options as a policymaker may not be so — and understanding what those constraints are can really influence how a study is designed. Now, I think it’s super important to allow for unexpected lessons to be learned, to allow for new ideas to come to the table, or new options that you may not have thought about in advance.

Karen Levy: But I’ll just make an example here. If a policymaker can only do something at the county level and can’t do it at a sub-county level, well, investing a lot of research energy into understanding heterogeneity within a county is literally not decision relevant. You may still want to do that for other reasons, but it’s much better to understand upfront the ways in which a particular policymaker is going to be able to wield the lessons learned. And so having those conversations I think can make for better research and can dramatically increase the likelihood that they will get used in the end. I think it can also help keep researchers accountable for delivering actionable evidence, and that’s important.

Rob Wiblin: Yeah. I guess we’re probably all familiar with the situation where it’s like, we go and get some information and it comes in one way — and then we realized that actually that wasn’t going to change the decision.

Karen Levy: That’s right.

Rob Wiblin: It’s interesting that this happens not just at the individual level — this happens at the organizational level, at the team level. It’s just an interesting phenomenon that we can put lots of effort into something that ultimately turns out not to be decision relevant. And so this precludes that possibility.

Karen Levy: That’s right. You know, we talk a lot about evidence for decision making, but we don’t often spend enough time learning about the decision making.

‘Sustainability’, and other myths in typical international development practice [00:20:45]

Rob Wiblin: You’ve been in the global health and development scene for a few decades now and have seen, I imagine, plenty of things in your time. If you’re game with me putting you on the spot, I would be excited to hear about any kind of common ideas or conventional wisdom in the field that you suspect might be misguided, or perhaps is barking up the wrong tree in some way. Is that all right?

Karen Levy: Sure. I’m always happy to share my very unorthodox views on some of these things.

Karen Levy: Yeah. So one of these concepts that I think we should interrogate a little bit is that of sustainability. And it’s not that sustainability is bad, but I think that insisting on sustainability — making it a prerequisite of support for a particular program or policy — can really distract us from what matters or what should matter.

Karen Levy: In some ways, I think that over-privileging sustainability in assessing the effectiveness or cost effectiveness or the extent to which we should prioritize a particular intervention can be almost like the moral equivalent of saying — to draw on Peter Singer’s famous analogy — “Well, a lot of kids drown in this pond, and since we don’t have a plan to save them indefinitely into the future, we shouldn’t save this one now.” And similarly, if I told you that for some reason, we were going to have to say, stop manufacturing a particular vaccine in five years, would we just stop giving that vaccine now? If that vaccine is truly effective and cost effective, well, then we should grab all of the impact that we can while we can.

Rob Wiblin: People who are in the development world will be familiar with this term, this kind of term of art, of sustainability. But for those who aren’t, do you want to kind of elaborate on what people mean when they say, “Is a program sustainable?”

Karen Levy: So what people usually mean is, “How is this program going to continue after whatever initial funding is given has ceased?” But unfortunately, I think what people often really are asking is, “How are you going to create a perpetual motion machine?” And in high school physics, I was taught that there’s no such thing as perpetual motion, so I feel like we’re kind of collectively trying to defy the laws of nature here.

Karen Levy: A lot of the problems that we seek to solve are going to require long-term sustained investment. So if a donor doesn’t want to or is not able to provide that support for a long period of time, that’s fine. But then I think really the conversation we should have is, “How are we going to shift the coverage of this to some other donor or some other source of funds?” and not, “How are we going to put in a little money, and then it’s going to create an engine that’s going to then fund itself?”

Rob Wiblin: From one point of view, for me, the supermarket is not a sustainable solution to the goal of feeding myself. Because if I stopped paying the supermarket, the supermarket would stop delivering me groceries, regrettably. But I’m okay with that, because I’m just going to keep paying for this thing, because it’s valuable for as long as I continue to want the service in return.

Karen Levy: Exactly.

Rob Wiblin: I guess you’re saying the case of a typical program in development is that if you want to keep delivering vaccines, you’ve got to pay for vaccines, and then you’ve got to pay to get them into arms and so on. And when people talk about the sustainability of a program, they kind of mean, “How are we going to pawn the cost of this program onto someone else, once we no longer want to be paying for it ourselves?”

Karen Levy: That’s right. I think we should be thinking about how long we think a particular program or a particular problem is going to require resources. Is this a permanent need? Is this something like basic education, which needs to be funded essentially indefinitely? Or is this something like deworming? I don’t want the Kenyan government to have to sustain a deworming program 20 years from now — I want them to not have a worm problem anymore, right?

Karen Levy: Is this something we think with sufficient investment now we could really move the needle on over the next five to 10 years? We may find that identifying a highly cost-effective solution — and funding it reliably for 10 or 15 years — is way more cost effective than constantly starting and stopping and reprioritizing and dabbling in various pilot programs that never get continued, because we have a little bit of magical thinking that they’re going to somehow just perpetuate themselves.

Karen Levy: But there are some examples where that can be done — there are some programs that can ultimately rely on market mechanisms. That’s terrific, and that can be a highly leveraged investment for philanthropic funds. But a lot of the work that we do, a lot of the problems that we seek to solve, require the provision of public goods and services that will not have market-based solutions — that are just going to require ongoing investment.

Rob Wiblin: I guess the natural thing that a lot of people might say — and sometimes I have this intuition as well — is that a program that a nonprofit is delivering would be a very natural fit for the government to be providing: that this is the kind of thing that would naturally be paid out of taxes, delivered by a government that’s elected. To what degree are you skeptical of the idea of passing programs onto the government in time? And is that a reasonable interpretation of the sustainability term or question?

Karen Levy: You’re absolutely right that very often the most common approach to sustainability is to seek government funding. Or the strategy is we’re going to get this started, and then we’re going to hand it over to the government. While I think that this is a good impulse — I do think that programs can and should work with and through governments — I think there are many aspects of working with and through governments, other than funding, that we need to prioritize.

Karen Levy: Governments are the agents of scale in society, so we should be thinking very seriously about and prioritizing government-led delivery, embedding programs in government policy, using government infrastructure. Those are all important. But the funding for some of these programs — above and beyond the cost of the infrastructure that the government brings to the table in the form of personnel or the existence of clinics or schools or whatever other society-level infrastructure that governments represent — in many ways, funding is the easiest thing for us to give, and the hardest for resource-constrained governments to provide.

Karen Levy: So I often remind people when they say, “Then the government will take this up” is that really what they’re saying is they’re just shifting the cost of the delivery from, say, the American taxpayer or from a wealthy philanthropist to the Kenyan taxpayer — who is ultimately responsible for the Kenyan government’s budget. And because the costs of getting something up and running and embedding it in the system are relatively high, we should be saying to governments, “If we can get this up and running together, we’ll help pay for some of those out-of-pocket costs for quite a long period of time.”

Karen Levy: We’re talking about resource-constrained governments that have lots of social challenges to face. The reality is they don’t just have big pots of money sitting around. It’s the equivalent of when we talk about GiveDirectly — like transferring money to poor people, right? We want to see a transfer of resources, and now there are highly impactful ways to do so.

Karen Levy: I think that if you have this model in your mind of like, “We’re going to get it started and the government’s going to take it over,” what you need to believe for that to make sense is that governments could or would do it, but they just need convincing or capacity building. I don’t think this gives them enough credit — I think mostly they need resources, right? And they need help in getting things from zero to one: technical assistance, support in building programs, advocating for them — this is all very valuable. But if we insist on only investing in that — and if we refuse to make contributions to the ongoing costs of delivery — well, then frankly, we’re just prioritizing supporting the activities that fund us and our work, and that aren’t actually truly sharing the burden of the costs of solving these problems in places that are poor.

Rob Wiblin: What do you mean by prioritizing things that fund us?

Karen Levy: If you think about the budget for a program that’s TA, technical assistance: we’re going to give advice to the government; they’re going to deliver the program. Well, that money goes for expensive foreign experts. It goes for meetings and conferences. It goes for the kinds of things that essentially foreign organizations or even local nonprofits can provide. But it does not actually help share the costs of actually delivering the goods and services and solving the very problems that we seek to solve as a community.

Karen Levy: So I would really like to see more funding with, through, and alongside government, and helping them to take a serious bite out of some of these problems. I want to see changes that show up in Max Roser‘s graphs on Our World in Data. And we can do this. This is within our reach. There are programs that are cheap and cost effective enough. We should be saying to governments, “We got this, OK? Work with us.” Not, “We’re going to do this for you,” but “We’re going to support this, so that you can do it without having to constantly advocate for that money to go here instead of there year after year.”

Karen Levy: I think the other thing is that there’s not a lot of understanding about how unfungible a lot of government funding actually is.

Rob Wiblin: Just to explain that idea: the concern might be that a donor ends up paying for a program that the government otherwise might have funded. So what you’ve done effectively is take, say, a $3 million program that the Kenyan government would’ve funded, and you’ve paid for it yourself. Now the Kenyan government has $3 million extra that it’s going to spend on something, and you don’t know exactly what — might be good or it might be ineffective. But because you’ve just — it’s this technical term, “funging” — kind of funged money from one place to another, it’s very hard to tell what counterfactual impact you’ve actually had. Sorry, go on.

Karen Levy: Yeah, so there may be cases where that happens. But I think the logical fallacy there lies in the reality that the vast proportion of government budgets are for recurrent costs, and mostly for things like salaries. So the amount of money that governments actually can have discretion over to put in one program or another is quite small.

Rob Wiblin: I see.

Karen Levy: It can be quite inflexible, and there are lots of different interests pulling on those funds — so it’s not entirely clear to me that funds would just shift from one place to another. And I think there’s a danger in, if someone who’s good enough at advocacy work could ramrod through a line item on a budget, what if the government said, “OK, we’re going to take that money from our malaria budget. Is that OK?”

Karen Levy: So again, I think we do have to think about what the counterfactual would be. But the enormous value that governments bring to the table for programs like this is the incredibly vast infrastructure they have — governments are the instruments of scale in society. And so the amount of funds that pay for activities, commodities, things that make programs work, if it’s a government-run program — like, for example, deworming often is — what they’re already paying for, which is all of the personnel that do the program, that’s what matters. And so really, what you’re asking is for them to prioritize tasking their people on a particular task — as opposed to shifting money.

Rob Wiblin: I see. Because that’s the asset that they uniquely have that you don’t have.

Karen Levy: Correct. That’s right. And creating an enabling policy environment in which programs like this can take place. Again, this is not to infantilize governments. In my experience, governments and the civil servants that I have had the privilege of working with are way higher capacity than people think. They are working within very realistic constraints that are hard for people in the private sector to understand.

Karen Levy: So even just redirecting funds — you don’t get to do that in a ministry. You make a budget, you ask the treasury for a budget, the treasury will approve a certain percent and may not even disperse the entire set of funds by the end of the year. If you don’t use those funds by the end of the year, they don’t get carried over: you have to send them back to the treasury. So you could have something line itemed, but if you don’t have the human capacity to actually deliver the program, the funds won’t get spent. So very often the constraints aren’t what we think they are. It’s not like there’s just this big pot of money, and someone’s deciding —

Rob Wiblin: They’re going from best to worst project, from the top to bottom. And then if you fund one of them, well, it just spills out into the next-best project.

Karen Levy: If only it happened that way. That’s right.

Rob Wiblin: It sounds like one thing you’re saying is that there’s a sufficient number of constraints here, and almost maybe like a randomness in what gets funded and doesn’t — that if you fund project X, there’s a pretty high chance that project X would not have happened if not for your money. That it’s not the case that it would’ve been funded otherwise — that’s the exception rather than the rule.

Karen Levy: I think that if you ask the right questions and look carefully about what is happening in a steady state — like when you engage in a particular place — then yes, absolutely.

‘Participatoriness’ [00:35:28]

Rob Wiblin: What’s another idea that’s common to hear that you think maybe people haven’t 100% thought through?

Karen Levy: I think another one is about participatoriness. Now again, just like with sustainability, I get where it’s coming from. I think that wanting to involve people in programs and policies that affect their lives makes a lot of sense. And there are people thinking about these kinds of issues in really sophisticated ways — for example, Tom Wein, who started the Dignity Project.

Karen Levy: But very often in practice, participatoriness ends up being user fees or cost sharing, in ways that are not particularly empowering to people. Think about if someone came along and said, “We’re going to fix the potholes on your street, and we need everybody to show up with a shovel.” There are these myths that people will appreciate it more if they help pay for it themselves.

Karen Levy: I feel like it’s a little paternalistic and a little insulting. People deserve services. They should be provided to them. They shouldn’t have to participate — they’re citizens of countries that should be doing these things for them. So I think that that term often gets used in ways that’s cynical, and when you really actually get down to, “Are you actually giving people decision-making authority?” — not so much. So yeah. I feel like that’s one we want to interrogate sometimes.

Rob Wiblin: Right. So ‘participatory’ sounds good — it’s a positively valenced word. What’s the steelman of participatory? I suppose for some people it’s going to sound appealing, because they’ll have heard about development projects that were brought by someone in Washington, DC who thought they were a good idea, and then they were imposed on people. And if only we’d listened more, then we would’ve realized that it was a bad idea. Or if only we’d included the community. Maybe that’s one reason that it sounds appealing. You’re looking like maybe I’m wrong about that.

Karen Levy: I mean, look, I don’t always know what’s best for me. And I don’t think people should have to know what’s best for them all the time. People will have different values, different things that they prioritize. Certainly when we’re talking about health, education, quality infrastructure — things that are about poverty alleviation and addressing some of the negative effects of poverty — I think we should be thinking more about being accountable to people. I think the word should be “accountability”: are we actually delivering to people effectively the goods and services that they deserve?

Rob Wiblin: And ideally, maybe from their point of view, it would be ideal for them if they didn’t have to participate — if it just happened.

Karen Levy: Exactly.

Rob Wiblin: They would like to be able to complain if things are bad, but they don’t want to have to show up with a shovel.

Karen Levy: That’s right.

Rob Wiblin: It’s another one that’s interesting, because being participatory sounds good. But I think if people were told that programs in their own life should be participatory, they might raise an eyebrow. If you said that provision of healthcare should be participatory, maybe I’m like, “No, I kind of want the national healthcare service to take care of this.” And the less they involve me, unless I am actually sick, the better.

Karen Levy: That’s right. It shows an extraordinary lack of value of people’s time, I think. So again, I’m not saying that people shouldn’t have a voice in their local governments, but that’s a different set of questions. So participating in making decisions about leadership, who then wield resources or make policies: absolutely. But participatoriness in the provision of basic goods and services in society, I feel like particularly poor people, we need to give them a break on that.

‘Holistic approaches’ [00:39:29]

Rob Wiblin: Yeah. I suppose feeling like you have to participate in something in order to make it go well, you would naturally construe that as a burden rather than a benefit probably. What’s another idea that you think people could think about a little bit more?

Karen Levy: Another concept that we hear a lot about in international development is this notion of ‘holistic approaches.’ So we want to get at the root causes of a problem — we want to solve the whole problem. Again, I really relate to the desire to really solve things. The problem with holisticness is, first of all, it’s very hard to know in big complex packages what’s actually working and what isn’t. And so throwing a whole bunch of things at something — some of them might be really good value for money and others may not.

Karen Levy: The other problem is that they’re really expensive, so we tend to see a lot of holistic programs that never scale beyond a pilot level. They can be sort of heartwarming and nice demonstration projects, but they never really go anywhere. And the landscape is littered with what I call these ‘perpetual pilot projects,’ which are doing wonderful things in, say, 20 schools. Well in Kenya alone, there are 25,000 primary schools, and they all have holistic needs. I think it can be hard for people from wealthy countries — and even wealthy people within poorer countries — to realize how much or how little money is actually spent on certain public goods. And that can really change your sense of what is realistic.

Karen Levy: So an example of this is I remember talking to an organization that was doing a school health program, and it was very holistic — it had a garden and training and it had a million bells and whistles — and it was a lovely program. And they wanted to talk about how it could be scaled up and made “sustainable.” I remember asking, “Well, what is the cost of this program per student per year?” And they were like, “Oh, it’s only $500 per student per year.” Well, at the time the entire national budget for education was $90 per student per year — that’s for everything: teacher salaries, school rooms, the whole nine yards. I think it’s a bit higher now, it’s maybe $140. So in what world is a $500-per-student-per-year program realistic? And then we come back to this, “Well the government will take it up.”

Rob Wiblin: Not yet.

Karen Levy: No it won’t. And they shouldn’t. And what we want is to do things that are actually going to be cost effective — so that if the government is spending money on it, or if anyone is spending money on it, we’re actually seeing impact and returns on that investment. So very often, holistic programs — as appealing as they may sound — just end up not delivering, frankly.

Rob Wiblin: Yeah. I guess ‘holistic’ is another positively valenced word that sounds good when you first hear it. There’s multiple different downsides to it though. One is that it is in strong tension with focus. And again, we all know in our own lives and our own organizations that it’s so much easier to really get something done properly when you’re focused on just this one thing — and you measure it, and you think at the end of the week, “Have I moved the needle on this thing?” — rather than having 10, 20 different metrics that you’re all trying to improve. It’s very easy to get lost.

Rob Wiblin: So I suppose that’s one problem. Another one you’re just saying is that you end up doing 20 different things. It’s quite expensive. Maybe a few of them are the things that actually matter, that actually are helping people. But because you’re doing them all at once, you can’t really figure out which ones those are. So improving the program is a lot more challenging.

Karen Levy: That’s right. And I also think, again, this can be connected to a desire to integrate things — to build government capacity. These are all worthy goals. But if you care about evidence, then you should care about requiring some proof about what actually moves the needle on that. And answering those questions might require a different set of activities than doing that through the guise of a school health program, for example.

Rob Wiblin: If I’m remembering correctly, I think the smallpox elimination program and the ongoing polio elimination program, they’re the opposite of holistic. They have this one focus, they have this one job, and that is kind of all that they’re trying to do. And as a result, they’ve been massive successes in a sense. So these people could be frustrated that they haven’t done more other things, but maybe realistically, if they’d had goals of eliminating five different diseases, they might have eliminated none, rather than kind of 1, 1.8, 1.9.

Karen Levy: Those are great examples. And one of the things that I’m most excited about when I think about programs like deworming is that they are now platforms upon which you could think about layering other things: getting one thing at scale working well in a cost-effective way, you now have a scaffolding and an infrastructure on which you can then layer other things. And then when you do so, you have a chance to do that really big at massive scale.

Karen Levy: I think that we don’t spend enough time thinking about the value of those distribution networks — I think that that’s undervalued. And we should often be thinking about interventions from the perspective of “What is the last-mile delivery mechanism, and how can we also use that to do other things?” rather than starting from a set of problems.

Karen Levy: Again, this is an area where a lot of people would disagree with me, but there’s a lot of talk in the neglected tropical disease space around integrating dealing with neglected tropical diseases. And my answer to that is always: “The fact that they’re all neglected is not a meaningful organizing principle. The fact that they all have long complicated names is not an organizing principle.”

Karen Levy: I remember having meetings with a group of experts here in Kenya and saying, “OK, we’re doing the school-based deworming program. What other neglected tropical diseases can we layer on top?” — really trying to lean into this, let’s integrate. And we went around the room and we talked through each and every one. And it was like, “Well, this one is in a different geographic location. And this one’s in dry areas and intestinal worms are in wet areas so that doesn’t work. And this one, you need to treat adults and not children. And that one’s children, not adults.”

Karen Levy: At the end of the day, even despite our best efforts, there was really no way to leverage the school-based deworming platform for those particular diseases. Now, there were other things that you could leverage the deworming platform for, but you have to sort of start with the delivery and then work backwards. So thinking about what should be integrated with what sometimes requires a little bit of outside-the-box thinking.

Rob Wiblin: Right. So I guess, what you might start with there is you say, “What we have built here is a system where once a year a teacher gives a student a tablet. What other tablet could they give them at the same time maybe?”

Karen Levy: Exactly.

Rob Wiblin: And that’s where you look for synergies.

Karen Levy: That’s right. And very often, if someone is looking for, “What implementing partner should we work with if we want to do a deworming program?” Well, I would rather work with an implementing partner that’s delivering something to schools — even if it’s not a health-related thing — than a partner that’s doing household door-to-door deworming through the health sector. It’s a different delivery. What you need to be good at in order to do that is just different.

Rob Wiblin: Yeah.

Karen Levy: And so the goal is not always what you need to have in common in order to be good at something. If someone’s got a great system in place for delivering something to schools, that’s an infrastructure that I can build on top of.

Rob Wiblin: Yeah. Maybe I’m flogging a dead horse here, but I feel with all of these, if we consider them in the context of our own lives and our own projects, it becomes clearer what the issue is here. If you’re someone who on a day-to-day basis goes to schools and then tries to get all the students to take a pill, and someone suggests, “Maybe we should also work on malaria or this other thing” — you actually think about, “Well, who’s moving where? Who’s saying what to whom at what time?” These very nuts-and-bolts issues.

Karen Levy: Right.

Rob Wiblin: And then you quickly see that actually the synergies that you kind of hope would be there might not be there. I don’t know, maybe you could make malaria work. But there could be other things where actually they don’t sync up, but I can imagine in the abstract it sounds good. The less you know about the nuts and bolts, the more it all sounds like, “This is kind of the same, right?”

Karen Levy: Right. And look, it makes sense that people would sort of organize themselves by the problems they care about solving. It makes sense that malaria people are going to want to talk to malaria people, right? But at the end of the day, impact is about execution and delivery — and what it takes to execute and deliver at scale is about that interface. It’s about that last-mile interface and what shape that takes: is it clinics, is it schools, is it households, is it urban areas, is it rural areas? Is it government personnel? Is it some other personnel? What is that mechanism?

Karen Levy: I would love to see a world where implementing organizations, development organizations come up and say, “We are problem agnostic, but what we focus on is being really good at household delivery (or whatever), so use us as the vehicle for reaching households for whatever it is.” And then of course, the other part of the puzzle is, is the pill effective? But you could have 10 different effective pills, and if you have no way to get them to the households, then what good is that?

Karen Levy: So really thinking carefully about delivery mechanisms, I think is a missing piece. There are organizations out there that are great at it, but all too often, they think about them as just a means to solve the particular problem they’re working on, and they might be leveraged in other really interesting ways.

Rob Wiblin: Yeah. We’ve been piling on a little bit here. I guess sometimes projects should be passed on to government. Sometimes participation by the community is appropriate. Sometimes there are a lot of synergies Between problems and they should be solved together.

Karen Levy: Sure.

Rob Wiblin: The thing is, I guess people can get a little bit beguiled by how nice these concepts sound, and maybe just end up overrating them, basically — even though they have their place.

Karen Levy: That’s right.

How the development industry sees evidence-based development [00:50:40]

Rob Wiblin: OK, pushing on. As we’ll get to, you’ve been part of the push towards evidence- and experimental-based development. I suppose I was very young in the ’90s, but my impression is that it’s kind of been gaining steam since the ’90s, when a lot of these experiments were first getting started.

Rob Wiblin: I feel I’m not part of it, but I’m kind of adjacent to it. My sense is that that approach has been through kind of a cycle of excitement and hope. Everyone being extremely keen about it — maybe a little bit naively so — and then there was a bit of a reaction, where some folks got disillusioned and people pointed out the weaknesses with it, and maybe things where it wasn’t as suitable to tackle.

Rob Wiblin: Where are we now with how the development industry sees evidence-based development? Do people have a balanced appreciation of both the strengths and weaknesses?

Karen Levy: Gosh. Well, far be it for me to speak for the whole industry. But I will say I can understand where some of the disappointment comes from. Because evidence alone is not enough, right? Just as we’ve been discussing, you have to take that evidence, you have to translate it into a deliverable program — something that is scalable. You have to fund it. You have to get the right partners together. You have to test it at scale.

Karen Levy: There’s a lot of things that need to happen in order to actually translate evidence into action, and this takes time. I think part of what we’re seeing now is programs that are based on the insights of studies done 10 years ago, 15 years ago are now actually really starting to move the needle in different ways. So I think we’re learning more about what it takes to kind of take that journey.

Karen Levy: Incentives for immunization is a really good example of this. There’s been evidence around this for a very long time, but you can’t just snap your fingers and say, “Here’s a really great paper, so why isn’t just everybody doing this?” The wheels move a little bit more slowly, and you have to really align a big set of partners — including governments — behind these ideas. But now, we’re seeing New Incentives doing spectacular things in Northern Nigeria. We’ve got IRD in Pakistan about to scale up an enormous incentives for immunization program. And it’s starting to take off in other places as well.

Karen Levy: The other thing that we’re seeing is we’re getting more sophisticated about what it is that we actually learn from RCTs. It’s not only like, “We’re going to test this little package and if it works, thumbs up. And if it doesn’t, thumbs down. And we’re going to just take this one little thing and deliver it.” What we’re starting to see is that we’re learning about the mechanisms behind interventions. We’re combining insights from different programs, different studies, and then using that to inform existing programs and new programs. And that’s a much more subtle undertaking.

Karen Levy: I think that the problems that we seek to solve as a community are long-standing ones: there are no silver bullets. When you think about RCTs in medicine — and even there it can take a long time — there is this sense that when it works, it works: you have a pill and you’re done.

Karen Levy: But we’re talking about complex social problems. And even if it is a pill, like in the case of deworming, it’s the actual delivery of it that is the challenge. Albendazole, deworming drugs, have been around for decades. That wasn’t the innovation here: it was the school-based delivery, mass drug administration approach — and then the impacts that that had on children and the externalities, et cetera — that was the true insight there. So similarly, we’ve got insights from behavioral economics, and the combination of studies together that’s now informing practice.

Rob Wiblin: Yeah.

Karen Levy: The other thing that I think we’re seeing is longer-term and more meaningful partnerships. So even within my experience in this field, in the early days we would send a grad student in to go talk to the minister and see if they… [laughs]

Karen Levy: I think we’ve gotten much more sophisticated about building deep, long-term partnerships with policymakers: co-creating research with them so that it can be more actionable, and then really walking alongside them as they take that journey towards building policies and programs. And so I guess I’ve always been a fan, but I hope that people who were skeptical will start to really see the impact that RCTs are having in global development now and in the years to come.

Initiatives in Africa that should be significantly curtailed [00:55:39]

Rob Wiblin: This next question might put you on the spot a fair bit, so I’m actually going to put it in the mouth of an audience member who submitted this one for you: “Given your experience, what are two or three aid-related initiatives in Africa funded by Western governments or others that you believe should be significantly curtailed or changed?”

Karen Levy: The first one that comes to mind are programs that require user fees. There’s been a lot of work on whether or not paying for something actually increases adoption. And there was a wonderful paper that Rachel Glennerster and Mary Ann Bates and others did called “The Price is Wrong,” which looks at what happens when you put in place even small user fees.

Karen Levy: The logic behind that always was that if people pay for something, they’ll value it more. And we’ve seen in practice is that, first of all, that’s not true: people do value and use things even when they are given them for free. But also that even small user fees can exclude the poorest and most marginalized — the people that you want to reach the most.

Karen Levy: And that the cost of levying those fees and collecting them can actually outweigh the value of those fees. So I think programs that have that — whether it’s in the name of participatoriness, or ownership, or cost sharing, or sustainability, or whatever the justification may be — I think that is very, very often misguided. And I would like to see people getting more free stuff. I think that would be a good thing. So that’s one.

Rob Wiblin: Yeah. Is that sort of pay-for-service or pay-something-for-service quite common?

Karen Levy: Yes. I think it is quite common.

Rob Wiblin: Interesting.

Karen Levy: I think sometimes it’s in the very same way that there are people who think we shouldn’t give poor people money because they’ll use it on cigarettes or alcohol. There is this sense that somehow people won’t value something unless they pay for it themselves.

Rob Wiblin: Yeah.

Karen Levy: I also think it comes from pressure from donors who want to see cost sharing.

Rob Wiblin: Right.

Karen Levy: It’s like taking on sustainability.

Rob Wiblin: They want to lever on people’s money. Yeah, I see.

Karen Levy: They want to leverage. Yeah.

Rob Wiblin: The story I’d heard on that one was that that was a common theory — that it was better to charge people something so that they’d value it and be more likely to use it — but that that had just been tested quite repeatedly, and it had been demonstrated that that was outweighed by other factors. That it was much more important to get as high uptake as you could, and even small fees had a huge influence on uptake. But it sounds like that hasn’t flowed through to all of the different programs where it’s potentially a relevant lesson.

Karen Levy: I think that many people have gotten that message. The bed nets programs have really taken that — I think that it’s had an enormous impact on how mosquito net distribution programs operate. But there’s a whole world out there of programs that are not influenced by these things, and so that is one that I still unfortunately see a little bit too often.

Rob Wiblin: It’s interesting. I’m just thinking about this in the context of the UK’s vaccination rollout, or the US vaccination rollout. Each dose, depending on the vaccine, costs between $5 and $20, something like that. But I think people knew intuitively, as well as from the research, that even if you charge people $2 for a vaccine dose, you would get a big drop in participation relative to it being free.

Rob Wiblin: And given the huge social value of people having vaccinations — hundreds, thousands of dollars even — that would be insane. I guess kind of the same principle applies here in other areas: if it’s really valuable for someone to use something, then charging them some nominal fee for it and having half of your population drop out is a really bad move.

Karen Levy: That’s right. And certainly in the case of vaccinations, not only should we not be charging people, but we should be incentivizing them.

Rob Wiblin: Paying them. Yeah.

Karen Levy: Right? We’re paying them to actually get vaccinated. But let’s take chlorine as an example. So how do you get people to chlorinate their water if they don’t have access to regularly chlorinated water? The theory was, if people buy chlorine, then they’re going to be more likely to use it. Well, it turns out that, first of all, it’s really hard to get people to spend money on essentially preventative health types of goods. But also, even if you give them chlorine for free, if it’s sitting in a bottle in their house, it sort of goes up on the shelf and you may not even remember to use it.

Karen Levy: Whereas the Dispensers for Safe Water program run by Evidence Action places chlorine dispensers at the point of collection. This makes the use of chlorine a social activity. So you see other people doing it, and that reminds you to do it. It’s part of the water collection process. That is what then generates uptake.

Karen Levy: Even the deworming program, the researchers on that study tried introducing very, very small user fees for the deworming tablets, and the demand dried up.

Rob Wiblin: It’s very interesting. The chlorinated water — where you provide the chlorine at the point where people are receiving the water at this common, I guess it’s wells or something like that?

Karen Levy: Yeah. Or springs.

Rob Wiblin: Springs that are shared. That raises another point in the design of a program: that it’s potentially much easier to have something fixed at the source, where you only have to change one tap — rather than fix it in everyone’s house, get everyone to do their behavior individually.

Rob Wiblin: I mean, we would think it was pretty nuts if each of us had to sanitize our water at home, based on something that we had to go and buy regularly in order to sanitize it. You just want to fix it at the source of the water, ideally. And the higher up you can do that sort of thing, the more you reduce the amount of people that have to think about something. And modernity is all about having things happen automatically, without having to think about them.

Karen Levy: That’s right. And they are considered public goods that just come with living in a society that can provide that. That’s what I want to see us achieving collectively: more and more of those things become just part of the landscape, what you get.

Rob Wiblin: Yeah. OK, so that is a program that potentially should be changed. Are there any promising funding opportunities that for some reason or other aren’t already getting funded by the big organizations? Anything that stands out?

Karen Levy: So there are a few areas that I think have a lot of potential, are very promising, that may become highly effective funding opportunities in the very near future.

Karen Levy: One of them is mental health. I think there are some really interesting and important new innovations in that space. And when we look at just the magnitude of DALYs around mental health problems, there is certainly a lot of work to be done there. Also, the more that we learn about how mental health challenges then have knock-on effects for other social issues that we care about — other health problems, economics — I think there are going to be a lot of highly cost-effective opportunities in mental health coming up.

Karen Levy: Another area that I think is underinvested in at this point is education. I think partly that’s because we don’t quite have the same tools yet that we do in health to be able to really measure and compare educational outcomes — and to really get at, in a very granular way, what the costs are and then what the benefits are. However, there’s a lot of really important work being done on this — there’s LAYS, L-A-Y-S, being done by David Evans and others in the education space. And so I think we’re going to just learn more and more about the impact and the benefit of educational oriented programs, and then the opportunities for funding there will increase.

Rob Wiblin: Yeah. We talked about LAYS — which stands for “learning-adjusted years of schooling” — in an interview with Rachel Glennerster a couple of years ago. It seems like a very useful innovation to create this standard measure — which is going to have problems, but nonetheless very useful to have some kind of standard measure of gain in actual education outcomes to compare across potentially quite different interventions, and get some sense of where they stand relative to one another.

Karen Levy: Absolutely. I also think that there will be some interesting opportunities in the education space because it’s an area where governments are spending a lot of money, mostly in the form of teachers’ salaries. So small interventions that can even make marginal improvements in educational outcomes are going to end up being really good value for money.

Karen Levy: Andy Zeitlin and some of his colleagues at Georgetown have been doing some work in Rwanda, looking at teacher incentives and connecting educational outcomes to parts of teacher pay. I think that has an enormous potential to be highly, highly cost effective, because it leverages the investments that the Rwandan government is already making in teachers’ salaries, and will help align the incentives of those teachers towards improving educational outcomes.

Misaligned incentives within the development industry [01:04:55]

Rob Wiblin: Nice. While we’re talking about ways the global development world could potentially be better, let’s maybe go through some concerns you’ve expressed over the years about the incentives that people in organizations within the industry are actually facing, and how they’re not always perfectly aligned with maximizing cost effectiveness.

Rob Wiblin: Of course, any industry organization has some of these sorts of issues; I feel slightly bad picking on the development sector or the nonprofit sector or the government sector as if they’re unique in this way. Of course, every organization, as we know, has its quirks and ways in which people don’t have the ideal incentives potentially. But what’s an example of a problematic incentive faced by people working in development that you think is important to keep in mind?

Karen Levy: I have been thinking a lot recently about how the structure of the business model of NGOs that work in development is really counterproductive in many ways to optimizing impact and cost effectiveness. We’ve come a long way as an industry in understanding that the percentage of overhead is not what matters, right? I think that is an issue that’s been extensively debated and most people who think in forward-thinking ways about this get it: it doesn’t matter if it’s 7% or 27% — what matters is, are you having impact?

Karen Levy: At the same time, as long as the business model is essentially that you’re funding an organization based on what’s pretty much a flat tax on your throughput, think about the perverse incentives that that creates in terms of efficiency, for example. A big threat to an NGO’s budget is underspending. Well, I would love NGOs to underspend, as long as they’re holding their impact constant, right?

Rob Wiblin: What do you mean by they’re kind of fundraising based on a tax? Do you mean that the organization overseeing it can only justify taking some cut of the total amount being spent on the program as a whole? So the more they find a way to actually spend on something — whether it’s good value for money or not — then that kind of justifies having the people at the head office, basically?

Karen Levy: An overhead rate is a tax, right? It’s a flat rate. So it doesn’t matter if that rate is 7% or 17% or 27%. What it means is that as the leader of an NGO, the way that you fund your core costs is by being bigger — not in terms of your impact, but in terms of your financial throughput.

Rob Wiblin: I see.

Karen Levy: I think this really creates some suboptimal decision making on the part of NGO leaders — they have to keep the lights on, right? If you are funding your core operations by taking a percentage of the money that you spend on charitable aims, then your goal is to just be spending more and more on charitable aims. And that’s not always consistent with what’s best in terms of impact.

Karen Levy: I had a conversation the other day with the executive director of a partner organization that I work with now through Fit For Purpose. They’re working in several different countries, and he was asking if we could help him out if they wanted to do some work in Kenya. Now, if I was a typical NGO and my business model was that for every dollar I put through my organization, I get a certain percentage to pay for my overhead costs, then subtly, my incentive is to want to say to him, “Oh yes, we can do that for you — run the money through me.” Because that’s not my business model, I can say to him, “Look, if it makes sense, if we’re the best people to do this, sure. But if not, actually, it might be better for me to introduce you to another organization that might serve you better in that case.”

Karen Levy: So I feel for leaders of NGOs. I have been in that position for much of my career, and it becomes very hard to separate the program as a goal from the organization as a goal. And we want these organizations to exist — I want there to be robust, healthy, vibrant NGOs that are great places for people to work. I don’t want those people to have to make decisions about how much money they’re spending, or how much money they’re raising, based on what they need to clear in order to pay their finance team in a particular year.

Rob Wiblin: Interesting. So the idea here is that you have some central nonprofit organization that’s running a whole lot of different programs on the ground. If they find a way to cut costs on those programs, or they find a program they think is not a great use of funding and they cut it, then it makes their overhead ratio look worse. Or if basically donors have agreed to fund some top-up percentage on top of the cost of program delivery to fund the central organization, then now basically their budget is cut — even though they’ve actually made a decision to use the funding more responsibly.

Karen Levy: Exactly. That’s right.

Rob Wiblin: Is that how donors actually fund the overhead costs these days? They basically impose it as a tax on top of the delivery?

Karen Levy: Very often it is a flat percentage. Many donors will have a cap, and so then there’s all these games about “What’s a direct cost? What’s an indirect cost?” — so that you can make it come under a certain threshold. I would really like to see both NGOs and the donors that support them look really clearly at the absolute numbers: What does it cost to run this organization? What are the fully loaded costs? What do we want the capacity of this organization to be, and what is it going to take in order to fund that?

Karen Levy: How else can we empower NGOs to tell us when something is not working and shut it down, or redirect funds in a different way, or say, “Look, we actually think we should wait a year — there’s an election, and this is going to be much more effective.” You don’t have the luxury of doing that when your core costs are covered by a flat rate percentage.

Karen Levy: I think that we’re moving in that direction — we do see more examples of donors covering core costs. But again, we’ve got to find new and creative ways as an industry to hold NGOs accountable for outcomes. But at the same time, not give them these weird moral hazard-y, perverse incentive types of situations.

Rob Wiblin: It reminds me, interestingly, of a model of fixed utility regulation that I encountered when I was an economist working in the government. So very often with an electricity grid, the actual cables that are connected to people’s houses to deliver electricity, the government wants to privatize it, hoping that a private company might be able to run it more efficiently. But then how would you get a private company? How do you reimburse them? Because there’s no competitive market for the utility; there’s no other cables that people could go to.

Rob Wiblin: So what they do is this “cost-plus” thing, where they say they’ll cover all of the costs that you’ve incurred plus some rate of profit, like 7% and all of that — which obviously then creates the incentive that the only way the utility can make more money is just to find some way to spend more money building up more and more infrastructure. So then you have to have a regulator that approves everything that they purchase in order to prevent them from inflating their costs. It’s an interesting tangle you’ve created trying to privatize it.

Karen Levy: Absolutely. It’s a tough problem. And I would love to see NGOs have some upside risk. If they’re able to do a project under budget and exceed the expectations with impact, well, they should be able to buy nicer office chairs, not fewer. In other words, I think that the definition of success needs to not be mutually exclusive with survival as an organization.

Karen Levy: I think there’s a lot of good thinking happening around this in the results-based financing space. Instiglio is an organization that is doing some really creative thinking around this. I would love to see more NGOs that are really confident in the fact that they can deliver something for a certain price, bid for things, deliver, and if they can come in under budget, they get to keep the balance. I think that would be a wonderful model. I would love to see more organizations willing to share that risk with donors.

Rob Wiblin: OK, so that’s a perverse incentive that donors are creating for other people. Do you have an example of a perverse incentive that donors face themselves?

Karen Levy: There are lots of examples here. I definitely think that there is an inherent incentive when you are working for a donor organization to want to come up with new ideas, new strategies, new programs. If you have a “set it and forget it” portfolio of programs that are working and they’re delivering impact, then the decision is really, “Should we keep funding this — yes or no?” Well, what’s the job for your program officer then?

Karen Levy: I remember in the early days when we were starting Evidence Action, at one point we made a trip and visited five or six donors in the UK. And I think every single one of them was in the middle of a strategy refresh. I remember feeling like, “All right, well, when you guys all figure out what you want to support, let me know.” The donors are in many ways the client: they pay for the work, and NGOs can only do work that they can get funding for. So I think donors sometimes underestimate the extent to which changes in strategy can really create incredible inefficiencies for organizations as they scramble to follow whatever it is that donors are funding. I think that we could probably all do a lot more good collectively if we made much longer-term bets that were a little bit less sexy.

Rob Wiblin: And you stuck with it.

Karen Levy: Yeah. Where you stuck with things over time.

Rob Wiblin: Yeah. The new staff member causing a review of the entire strategy. It’s always so ambiguous: Do we need to do a strategy review? Should we change the strategy? These questions are unanswerable. I guess sometimes you really know that things are going wrong, but very often you’re like, could it be better? Should we review all of this stuff? It’s almost always a plausible thing that you could do. And then potentially the chaos that you’re creating downstream — or the fact that your current strategy has problems, but your new strategy will probably have other similarly important problems — it’s hard to see.

Karen Levy: That’s right. And especially if you are a donor, you are having your impact through others. So you really have to think about how what it is you’re funding — or even how you communicate about what you’re funding — essentially creates the market for a particular set of programs that are going to then respond to the availability of funding.

Rob Wiblin: I’ve heard you say before that people should reassess how good it is to be boring. That boring is underrated; boring is potentially its own strategy that other people aren’t following, where you can get an edge by trying not to be sexy. Do you want to elaborate on that?

Karen Levy: So it is inherently more interesting to fund innovation, to invent new things, to catalyze things. When we think about the problems that we’re trying to solve in society, we don’t want people to have to think about these things every day. It should be the background noise to life. And I would say that the unit costs of delivering a particular good or service go down enormously, the more boring it is to deliver it. If you can make something so mundane that it doesn’t really require a lot of thinking, well, suddenly that is actually incredibly scalable.

Karen Levy: There are a lot of innovations that are on the shelf that are not yet delivered at the scale and scope that they might be. Maybe this is a partial answer to your previous question of what else should we be funding? I think we should probably be funding more of what we’re already doing, before we take on 10 new things that we can’t fulfill their full promise because there aren’t enough resources to do that yet.

Rob Wiblin: It’s interesting. Obviously there’s a natural case in favor of innovation, which is that maybe you’ll find some new drug that’s incredible that will transform things. Or maybe you’ll find some way to dramatically improve education, as you were saying — and the education budget is so large that a 10% increase in efficiency would be a massive deal. So I guess it’s not the case that innovation isn’t a good idea, but I suppose —

Karen Levy: We need both.

Rob Wiblin: Yeah. I guess one thing to keep in mind is that the total budget available for all of these development projects, globally, isn’t that large, and the amount that might go into delivering your innovation might not be that large either. One has to keep in mind the ratio between how much we’re spending on innovation and improving things, versus the plausible maximum budget that this is going to attract in coming decades — and just ensuring that that ratio makes sense. Is that right?

Karen Levy: Yeah. If you really think about it from an investment perspective, sure, you want to spend a little bit on trying a lot of things. But then you want to cull really quickly — so that you can then home in on the things that are really viable, that’ll have legs.

Karen Levy: I would also say that every time there’s an innovation that is not meeting its potential for scale, we’re losing really important opportunities to learn how to select the next innovation — because you learn a lot about what is viable, what is feasible, what is scalable, what is impactful? Things change when they move from pilot to scale.

Karen Levy: One of my favorite images, I wish I had thought of it myself, was on some presentation I saw somewhere. The image was a baby. It was talking about “What is scaling?” A human growing up is not a baby turning into a bigger baby — it’s a baby turning into an adult. And an adult is a different thing than a baby. It’s the same thing with programs. We’ve learned a lot about innovation, and we should keep doing that, and it’s incredibly important that that gets funded. But in the absence of really solid paths to scale for viable ideas, we’re missing the opportunity to learn which innovations will work better and also learn how to scale them more effectively. We’re also losing the opportunity to layer new innovations on top of robust delivery platforms that previous innovations have put in place.

Rob Wiblin: That are already working.

Karen Levy: Yeah.

Deworming: the early days [01:20:17]

Rob Wiblin: With that in mind, let’s wind back maybe to the early 2000s. As I mentioned in the intro, you were pretty central to the big push in global health to get hundreds of millions of kids dewormed on a regular basis. It sounds like it’s a really exciting story that might be able to offer up a whole lot of lessons on how to get things done in global health today, as well as just more generally, and it’s a story I’ve only heard little fragments of. So if it’s all right with you, I’d be keen to go through it in a little bit of detail. How did you end up establishing the Innovations for Poverty Action (IPA) office in Kenya?

Karen Levy: So I will say it was one of these incredibly lucky accidents that really just takes your life in a different direction. I had just come back to Kenya from the UK. I was working on my PhD, and I was doing a little bit of consulting for a few different organizations. And someone suggested that I call an acquaintance of mine, a guy named Chip Bury, who was the Regional Director of an organization called ICS, International Child Support. This was a Dutch organization that worked in Western Kenya doing a range of programs, and they had a few research projects that they were also working on.

Karen Levy: So Chip needed some help doing some strategic planning. The issue at the time was they were doing these studies with these professors in the US, who I had never heard of at the time. And over the years, the research projects had grown incredibly and were sort of starting to overwhelm their core programs — there were some debates about how they were sharing costs and some challenges. And so he had asked me to come in and help them do a strategic refresh.

Karen Levy: So I came in and I looked through the work that they were doing. I had just finished a master’s and started a PhD in development planning, and I had spent many years on the coast in Kenya, starting an NGO and doing some community-based work. So I was really a pretty classic traditional development worker, and always sort of felt good about the work I was doing. But to be honest, I had questions about whether we were really having any deep impact, and why all these projects were really small.

Karen Levy: So I started to look through the work that this NGO was doing in Western Kenya, and I saw some of the working papers that these researchers had started to develop — one of which was the deworming paper, which was done by Michael Kremer and Ted Miguel, based on this RCT they had done in and around Busia doing school-based deworming.

Rob Wiblin: When was this? What year?

Karen Levy: This was in 2005 that I was doing this consultancy, and the study had been done in the late ’90s. And it may have just been published at that point, but I think I really was seeing a working paper. Now I am not an economist, so I didn’t understand a lot of the details of the paper, but I got the gist of what they were talking about. And I saw that this group of people had this rigorous approach to measuring the impact of these programs in a way that I had never seen before.

Karen Levy: I remember saying to Chip, “This is the most exciting stuff I’ve seen in a really long time, and not only should you not get rid of these research projects, you should double down on this. We should really make a strategic plan that’s about centering everything on these research projects. Forget that other stuff that you guys are doing — let’s do this.” And I remember saying at one point, “Gosh, we could even deworm all of Nairobi” — as if, who could think even bigger than that, right?

Karen Levy: So in the course of that work, I met Michael and Esther Duflo and Pascaline Dupas and Ted Miguel and others who were coming to Kenya regularly — and listened to them talk about their work, and helped to try and negotiate some arrangements about how they were going to share costs. I put together this presentation that Chip was going to then bring to the Netherlands to present to the board, and I remember making all of these charts going up and to the right about how this was all going to grow and be spectacular. So he goes off to the Netherlands and he presents this to the board and he comes back —

Rob Wiblin: The board of the organization you’d been working for?

Karen Levy: Yes, the ICS Board of Directors in the Netherlands. He comes back to Nairobi and he calls me in. And I said, “So how did it go?” He says, “Well, they didn’t buy it, and I’m not allowed to hire you anymore.” And I remember just being so deflated. This was relatively early in my career, and I remember thinking, “Oh gosh, maybe one shouldn’t speak truth to power.” I don’t know, you really start to question yourself. So I said, “OK, I’m sorry,” and then I sort of went off and did some other things.

Karen Levy: Sure enough, a couple months later Chip called me. And he really understood and valued these research projects — he was torn himself. So he calls me back a couple months later and he says, “So you remember that guy Michael Kremer that you met during this project?” I said, “Of course, yes.” “He’s going to call you.” I said, “OK, all right.” So Michael calls me, and it had essentially transpired that this NGO had decided that they did not want anything to do with these research projects anymore and they wanted to focus on child sponsorship — the work that they had been doing. And I remember one of the phrases: “We don’t want to study people — we just want to help them.”

Rob Wiblin: It’s one or the other, it’s one or the other, Karen.

Karen Levy: Right? Yeah. That was their call to make. So Michael called me, and I think he had seen that I appreciated the work that they were trying to do — that I saw the value, that I was excited about it — and also that I had some experience with organizational development in Kenya, having started an NGO, and that I had some management experience from that perspective. He basically asked me to take on what, at the time, was a six-day consultancy to try and help him figure out what to do with these projects.

Karen Levy: They had a few research projects that were ongoing — these were RCTs running in Western Kenya through this organization. So now what were they going to do with these research projects? Should they find another organization to work with? Should they start their own? If they start their own, should it be based in Kenya? Should it be based in the US? So to sort of help him figure out the pros and cons of each of these options to help with this transition.

Karen Levy: And then almost at the very end of the phone call, he says, “Oh, and by the way, you should probably also talk to Dean Karlan at Yale. He’s started this thing called IPA. They’re working in Peru and the Philippines, and maybe he’ll have something to say.” Now I thought Dean Karlan was a dean at Yale, so I’m looking through the Yale website trying to find a dean named Professor Karlan. So that’s how I met Dean, and I learned about the work that they were doing, and how they were building the capacity to run RCTs in these other countries.

Karen Levy: So I came back to Michael after the end of these six days, and said to him, “Here are my suggestions. If you know and trust this guy Dean, they are doing exactly the same kind of work that you are. You could share back-office functions and you should go ahead and open up the Kenya branch. And if not that, do this, if not that…” — I gave him a sort of series of options. And that was it. To me, I was like, “Job done,” right?

Rob Wiblin: That’s my six days.

Karen Levy: Right. I’ve done my work. And so Michael — being Michael, the amazing person he is — he calls me back a couple days later and says, “So could you do that?” And I said, “Do what, Michael?” He says, “Help us get the IPA office set up.” So I said sure, and again was working as a consultant. I said, “Let me figure out what you guys need to do,” submitted some registration papers, and I said, “Well, you’re going to need a bank account.” So I wanted to go and talk to some banks about getting them an account. And of course, if you walk into a bank and say, “I’m a consultant,” they don’t want to talk to you. You need to be a representative of the organization.

Karen Levy: So I called Dean and I was like, “Look, I’m not really precious about this, but you need to give me a title. I’m just going to get 50 business cards printed. I just need to be able to have some conversations on your behalf.” And so he says, “I don’t know, what do you want to call yourself?” I said, “Whatever you think is fair.” He said, “How about Country Director?” So that sounds quite nice, and that’s literally how I became the Country Director of IPA Kenya.

Karen Levy: It was the most incredible stroke of luck. I ended up with a front-row seat and a participatory role in just the most extraordinary and inspiring movement of thinkers and doers that I could have dreamed of. And I spent many years learning from and working with some of the very best minds in this field and participating in the execution of these RCTs.

Rob Wiblin: So this was a time when RCTs were getting a lot of traction, people were very excited about it, a lot of really talented people were joining the field. Tell us about that, about kind of the mood in the room.

Karen Levy: I absolutely feel like there was this sense that we were onto something really exciting, and many of the people who came through Busia in those days as graduate students and as research assistants are now professors and practitioners in their own right. And at the same time, we couldn’t have dreamed that it would’ve had the impact on the field beyond research. I mean, I remember early conversations with Michael, and asking him, “How big a machine are we building here? How many studies do you think will be running five years from now, 10 years from now?” And I remember him saying at one point, “Two or three, maybe” — so I don’t think any of us realized quite how big it would become.

Karen Levy: Another thing that I think was important about those early days is because the idea was still very new, it wasn’t as easy or obvious to have implementing partners to work with. So in a lot of cases, we were actually delivering the programs that we were studying, and as a result learned a lot about program design and delivery. Now, ideally you want to be working with partners who are doing the delivery and those roles are really separated, but I think there were a lot of advantages to really diving in and getting our hands dirty and doing a lot of experimentation as we went.

Rob Wiblin: I guess doing it yourself slowed things down, but at the same time, you really learned exactly what was going right and what was going wrong, and I guess it puts you in a much better position to manage projects like that in future.

Karen Levy: That’s right. The other thing is that a randomized controlled trial needs to be tightly controlled, right? You need to be able to ensure that the randomization is done well. For early-stage RCTs, where you’re trying to test a proof of concept, if something doesn’t work, you don’t want to be asking, “Well, did it not work because it’s not a viable idea, or did it not work because it wasn’t well delivered?” So you actually want to have quite tight control over the quality of delivery, so you can really isolate the research questions that you want to answer.

Karen Levy: Of course, as things grow, that’s exactly the opposite: you want to take your hands off the wheel and see “Does this work when it is delivered as it would be in steady state?” — but that’s a question further along the value chain, really.

Rob Wiblin: Yeah. So it sounds like this whole thing was to some extent inspired by that famous deworming paper, which was done in Kenya. Michael Kremer was one of the authors; there were some others as well. But that’s what prompted the creation of IPA and this setting things up in Kenya. Is that right?

Karen Levy: So the deworming study was certainly one of the earliest, and one of the studies that then inspired lots of other work, but it certainly wasn’t the only one. There were a number of studies that were being done in Kenya, but also elsewhere: in Peru, in the Philippines, in Ghana. The network of PIs — of researchers that were building the methodology and exploring different sectors of development — was a much, much broader effort that was taking place during that time.

The problem of deworming [01:33:36]

Rob Wiblin: So we’re about to dive into this story of scaling deworming efforts. I think there’ll be some listeners who’ll be very familiar with deworming, like what it’s all about, how much it costs, what problem it’s trying to solve. But some other people won’t have read a whole lot about deworming, which would be very understandable. So maybe let’s do a bit of a refresher of the basics. What problem is deworming designed to address?

Karen Levy: So about two billion people in the world live in places where they are susceptible to regular chronic intestinal worm infections. These infections sap your energy, they can make you anemic, they can make you more susceptible to other diseases. And intestinal worm infections are particularly intense in children, and are therefore a particular problem for kids. They compromise children’s nutritional quality at a time when they need that nutrition most.

Karen Levy: Unlike acute diseases, which are very obvious and clearly need to be addressed, worm infections are often considered a symptom of childhood. They’re so pervasive that people often don’t really think of them as something that needs to be addressed.

Rob Wiblin: I was talking to a parent here in London the other day, and she was saying, “Obviously my child just always has colds, just constantly has colds. I always have colds because they’re going to daycare with other kids. They have no sense of hygiene.” I guess it’s basically the same idea.

Karen Levy: Yes. But imagine that having that cold then meant that — particularly if you’re already nutritionally compromised — now that’s making that even worse, and it’s going on and on and on. You’re missing lots of school, and so the cumulative effects of worm infections in individuals and populations over time can really be extremely costly.

Karen Levy: Now, at the same time, there is this very simple solution, which is a deworming tablet. That technology has existed for a very long time. But because it’s not an acute disease, and so perhaps not a priority — and also because the diagnosis of having worms is more complicated and expensive than the treatment — deworming had not really been a regular part of health packages.

Rob Wiblin: So an interesting thing is that this tablet costs under 50 cents per person. But I guess you’re saying it didn’t fit naturally within the healthcare system, where you would have a serious problem and you’d go and talk to a doctor, and the doctor would diagnose you and then the doctor would treat you. Basically, you kind of want to skip all of that because the pill costs under 50 cents and it doesn’t really cause any serious side effects anyway. So you may as well just take the pill on a regular basis, even if you don’t know if you have worms. But the whole profession doesn’t have much of a system for delivering medicine of that kind.

Karen Levy: Well remember, we’re talking about places where going to doctors and getting diagnoses is not happening as we want it to be anyway.

Rob Wiblin: Yeah.

Karen Levy: Right. But absolutely a big insight around deworming is the value of mass drug administration — where if you have prevalence rates over a certain level in a particular area, it just makes more sense to treat everybody than it does to figure out individually who to treat. But even that was not new knowledge.

Karen Levy: I think what the real innovation and the new insight from this randomized controlled trial that Michael and Ted did was, first of all, it used the school infrastructure to target children — traditionally mass drug administration had been a door-to-door activity. I think coverage rates are lower. Health facilities are less pervasive than schools — you’re much more likely to live near a school than a clinic. And as primary school enrollment has been increasing over time, school is where the kids are, right? So the most efficient and easy way to treat all the kids is to do it at school.

Karen Levy: The other lesson that was really important was about the positive externalities. It’s very easy to actually underestimate the impact of deworming if you’re only looking at the kids that you treat, if you’re treating individuals. So a bunch of kids not having worms is actually good for the kids around them, even those who aren’t treated, because they’re less likely to be infected with worms from their peers. And so they found that the impacts were much larger than one might expect.

Karen Levy: The other thing that they saw was the impact on school attendance. It was a new thing to think of this as an education intervention, not just a health intervention. And so all of those factors together really helped to make an argument for the cost effectiveness of school-based mass drug administration for worms.

Rob Wiblin: So this RCT of giving deworming drugs to all of these kids in these schools, as I understand it, it was looking at school attendance, it was looking at maybe test performance, and then they also did some follow up later on when people were working, to see whether their earnings had been increased in the long run. And basically, the results were kind of extraordinarily positive — so positive that I think people always would’ve suspected that if you went and studied it some more in other places and other times, that you probably wouldn’t get a repeat of something so amazing. But nonetheless very exciting, so that a lot of people wanted to get on board and test this again and scale it up.

Karen Levy: The really incredible work that Ted Miguel and his colleagues have done is in following up on this cohort of kids who are now adults. This is how he’s been able to observe these wage effects so many years later. That’s a very unusual and special longitudinal look at the impact of an intervention like this — a seemingly very small thing that has very large knock-on effects over time.

Rob Wiblin: Yeah. We’ll stick up a link to a new 2021 paper, which Kremer and some other co-authors published, where they are following up with these kids decades later. I guess that they’d now be in their early 30s or something like that, and I think they’ve got records of their income for over 80% of them. And they’re finding that it seems like being randomized into the group that got deworming tablets is still increasing their income by a pretty large amount today.

Karen Levy: That’s right.

Rob Wiblin: So we’ve had something like 20 years since this trial was first wrapped up, or people might have had a sense that the results were going to be very positive. Do you have an idea of, at a full global level — out of all of the two billion people who are vulnerable to intestinal parasites of this kind — roughly how many now are getting deworming tablets on a regular basis, who otherwise might not have if not for this work?

Karen Levy: It is in the hundreds of millions. Of the two billion people who live in intestinal worm endemic areas, it’s about 800 million children that we’re talking about. India alone is now deworming I think 200 million children every year. So the numbers are really vast, and more excitingly, we are now starting to see evidence of a reduction in prevalence of infections and also in the intensity of those infections.

Rob Wiblin: So we’re potentially helping to eliminate them in the population more generally, because people are getting these tablets sufficiently often that the chain of transmission is getting weakened.

Karen Levy: That’s right. And the goal is to essentially eliminate worms as a public health problem, right? So you may not eradicate worms completely, but you can get infection rates down to a point where it is not a pervasive public health problem.

Rob Wiblin: Some listeners will have heard there’s been this active debate during the 2010s among economists and among health folks — and also, I guess to some extent, between economists and health folks — about how much benefit is provided by deworming kids, because there’s been so many followup studies to this Miguel/Kremer study.

Rob Wiblin: This episode isn’t primarily about that issue — if we wanted to do that, we should get someone on who’s embedded in the really technical questions of the papers here. But a lot of listeners will have heard of that debate. So in brief, do you have any kind of overall take on that? I hate this expression, because it is just a polite disagreement among academics, but people have called this the “worm wars,” I guess for fun.

Karen Levy: The worm wars. So look, I think it’s great to be skeptical of results, and I think we should always be open to critique and updating our priors as we reassess evidence. At the same time, my point of view on this is very consistent with GiveWell’s, which to me makes a lot of sense: even if you think that there’s a very small chance that the results are accurate, or that they are of the magnitude that we think they are, this is still an incredibly good buy. I see things from the point of view of what’s relevant to real decision-makers, right? If someone comes along and says, “I have a way better thing for you to do instead,” I’m certainly open to changing my mind.

Karen Levy: But deworming programs are incredibly inexpensive. They create a platform upon which you can layer other interventions in really valuable ways. We’re seeing a reduction in the prevalence of worms. The individual diagnosis of worms is extremely expensive and doesn’t seem like a good use of time. When I think about looking at the evidence, how policymakers should make a call about this — to me you always end up on the side of let’s do these programs.

Rob Wiblin: Yeah. Makes sense. So the argument there is an expected value one: that we’re not sure exactly what the size of the effect is, but there’s a high chance that it’s a small positive effect, some chance that it’s a really large positive effect, and it costs so little that in terms of expected benefits over expected costs, it still looks really excellent.

Karen Levy: That’s right. When you think about the deworming program in Kenya, I think it costs somewhere around $2 million a year. I mean, this is like the line item for security for some programs in some countries, right? I wish that we subjected some of those programs to a fraction of the scrutiny that we have subjected deworming to.

Rob Wiblin: Yeah. I think lots of people might agree that this is above the bar to do. I suppose it’s almost a victim of its own success, where people were so excited that they started saying that this is like the best thing — and that’s a much more specific and intense claim. So it’s natural that people might respond to that and say, “Well, is it? It’s hard to be sure about that.”

Karen Levy: I don’t think anyone serious ever thought that deworming was a silver bullet — that it was going to solve everything, that it was the only thing we need to do, or even the best thing to do. Is it a really good value for money, worthwhile thing to do? For me the answer to that is clearly yes.

Rob Wiblin: Well, we’ll stick up some links to some worm wars blog posts, and I guess give voice to people who want to explore that one more.

Deworm the World [01:44:51]

Rob Wiblin: Let’s push on and talk about Deworm the World, and the efforts that you were engaged in to actually reach millions and millions of children in Kenya, and then demonstrate something that could be used elsewhere as well. What did you and your colleagues concretely do to try to get more people dewormed?

Karen Levy: So I was working as the IPA Country Director, and I was always very interested in how we were going to take the results of these studies and actually influence policy with them — and that was very much Michael and Ted and Esther’s agenda as well. The Deworm the World initiative was actually started by Michael and Esther at the Young Global Leaders Forum at the World Economic Forum, I believe in 2007 — where they were on a committee looking at education, and suggested that that committee take on deworming as their agenda. That is how Deworm the World was founded.

Karen Levy: And the decision was made to start in Kenya because the study had been done here, it was clearly a problem here, and we had people on the ground who were interested in seeing it happen. So I started to engage with people in the ministries of education and health to explore the possibilities of doing a deworming program here.

Karen Levy: Now, I was not the first one to have these conversations: there were people at PCD (the Partnership for Child Development) in London who had been working on this issue, and people here at the World Bank who were aware of the study findings, and had actually put some money for a deworming program into a budget of education-sector support that was coming to the Kenyan government. So there were a lot of pieces that were falling into place, but at the same time, there wasn’t yet a deworming program.

Karen Levy: And governments have a lot of priorities — they have a lot of things that they need to do. There was and is a School Health Unit within the Ministry of Education and School Health Unit in the Ministry of Health — and they had a very long agenda. And there was no one person who woke up in the morning and their sole agenda was to make sure that there was a deworming program in Kenya. So that sort of became what I woke up in the morning caring about.

Karen Levy: We had a lot to start with. There was the experience of this study — so people who had actually done the school-based deworming program in the 75 schools that had been part of the study. And there were also people at KEMRI (the Kenya Medical Research Institute) who had done a pilot program in a district called Mwea — they had developed a teacher training manual. So there were some basic bits and pieces that were the parts of a program, but they were small. That pilot was, I don’t know, maybe 40 schools.

Karen Levy: So what we did was we sat down with partners in the government to think about and design: “What would it look like to do this program at a national scale?” And there were a number of different pieces that needed to come together. First of all, we needed to ensure that the policy and regulatory environment was in place that would allow for this. There was a school health policy that was still in draft form that hadn’t been finalized, so we worked with the government to help make sure that got finalized.

Rob Wiblin: So this is kind of changing the regulations of what the school is permitted to do, such that this isn’t allowed or actively encouraged?

Karen Levy: Yeah. So first of all, school-based deworming mass drug administration in schools needs to be part of the policy. Teachers need to be allowed to disperse a drug in schools. You need to be able to store deworming tablets at a school. There’s all kinds of rules that need to exist in order to be able to do this at a massive scale, and so we worked through that. But alongside that, we had to figure out the delivery model. So we have a few million deworming tablets sitting in a warehouse. How do you get from that to several million children around the country actually getting dewormed?

Karen Levy: I remember sitting at a whiteboard, one of my very favorite places to be, with a group of people from KEMRI and from the teams that had done this pilot program in Mwea. I was asking them about the different steps that they had undertaken, because the whole process of scaling is about standardizing what the process is. So I remember asking, “Well, how many times do you need to go to each school?” And of course the answer to that question is always “it depends” when you’re doing a pilot. And they said, “Well, some teachers, they got it. So we only went there once. And other places we had to go…” — well, this is not a recipe. That’s fine when you’re doing it in 40 schools, but not when you’re wanting to do it in 8,000 schools.

Karen Levy: So when we really pushed up against that, and really pulled out what were the essential elements of this program that you needed, it turned out that actually the answer to that question was zero: you don’t need to visit the school at all. What you can do is bring together two teachers from a set of schools, train them, and distribute the drugs and all of the forms and everything to them — and that made that last mile an order of magnitude more efficient. And so we built a cascade that would deliver training, drugs, forms, et cetera, from the center down to the school level, in a series of steps, in a way that we hoped would maximize efficiency, but would maintain fidelity to design.

Rob Wiblin: So you were there having conversations with people who’d done this kind of a test scale-up to 40 schools. Were you just going ahead and designing how this scheme would work at a national level before the Department of Education or the Department of Health had said, “Yes, we’re going to do it.” Was this part of making the pitch to them that it was viable?

Karen Levy: It was absolutely a collaborative effort. I think there was a lot of interest in doing it, and we worked on generating and maintaining that political will. Michael and Esther were able to get Raila Odinga, who was the Deputy President at the time, to make a speech at Davos claiming support for the program. People could see the value of it, but even interest and political will doesn’t necessarily translate to boots on the ground and all of those specific plans. And so sometimes it was things as simple as developing a high-quality, macro-enabled Excel spreadsheet that made it really easy to calculate what it was going to cost in each district, depending on attributes of that particular district — how many schools there were, how many kids, et cetera. This really made it viable for people in the ministry to take it on.

Rob Wiblin: Interesting. So one change you made when you were going from 40 schools to thousands and thousands of schools is you just bring one teacher from every school together to do training for a day or two or something like that, or maybe you do it at the province level, and then they all go back and do it. Were there any other major changes that you made as part of thinking about how you would reach 100 times as many places?

Karen Levy: Absolutely. So first of all, we had to design this overall cascade in a really thoughtful way. If you have too many steps, there could be too much erosion and too much time can go by as things move along. On the other hand, if there are too few steps, the growth is too big in each step, so you’ve got too many people in the room and you can’t train them all. So we did some experimentation with that.

Karen Levy: I will say in the early years of the program, we delivered it in a series of waves. So we had a group of master trainers that would set off the program in one particular area and then move on to the next, and so it was like a cascade happening in stages over time. Now this is not actually an optimal way to do a program like this. Big programs like this often work better as a big one-off campaign, where you have a “deworming day.” But to get to that point, you need to try it a few times in order to get it right. So doing some successive waves allowed for that adaptation in a way that doing something once annually doesn’t necessarily allow for.

Karen Levy: The other kinds of things that we had to think about — and that are very different than when you’re doing a small program — is what kinds of community sensitization activities are needed? How are we going to make sure that the community knows that this is happening, that the drugs are free, that they’re safe, that they should bring their kids to school that day? That even kids that are not enrolled in school can show up and get treated for free? Thinking about it as a whole package is a very different prospect.

Karen Levy: Another thing is you want to plan it such that you don’t need to adapt to changes in the context — you want to make something that is going to work as best as it can in the very most places. And so again, that required a little bit of trial and error and some learning over time.

Where the majority of the work was happening [01:54:47]

Rob Wiblin: Where was the majority of the actual work happening here among the deworming folks? It seems like, so you’ve got to distribute ideally six million pills. I guess most of the actual hours are being put in by teachers and administrators at the schools. So for you, you’ve got to procure the drugs. You’ve got to do training for one person from each school. You’ve got to make sure that the drugs are getting out to places in time. What’s the actual full-time equivalence ending up here?

Karen Levy: So what the Deworm the World team did, and still does, is support the government as they deliver the program. So certainly in the beginning, a lot of that work was around design. And we are not even training the teachers: we’re training the people who are training the people who are… There’s several steps in the cascade, but also it’s not even us doing the training — it’s bringing together experts from the Ministries of Health and Education who can do that, putting in place the protocols, figuring out how to figure out how many tablets need to go to each school, and are they packaged in increments that make sense so that they can be delivered efficiently?

Karen Levy: The other thing that we were doing, and that the Deworm the World Initiative (which is now at Evidence Action) still does, is really support the monitoring and evaluation of the program. There are a number of different ways that that program is monitored. There is process monitoring, which looks at: are the activities happening with sufficient quality and fidelity to design? There’s also monitoring the administrative data that’s generated by the program: so the coverage numbers that we’re seeing, do they seem accurate? And then there was work being done by KEMRI to actually look at worm prevalence rates before and after treatment to see if we were seeing a reduction in prevalence and intensity: were kids actually getting dewormed?

Karen Levy: And so we really worked on putting together a monitoring and evaluation approach that really looked at the whole theory of change, and helped to make sure that things were happening along the way. Especially in the early days, that generated a lot of lessons that were fed directly back into program design.

Rob Wiblin: Yeah. What’s an example of that?

Karen Levy: An example of that is the thing that really made a difference about whether or not kids would get dewormed: once the school had the training and they had the tablets, they did pretty well. The big danger was them missing out on going to that training and getting the drugs in the first place. We found that although all schools were eligible to participate, private schools were getting left out — not for any reason or malicious purpose, but because district local-level education officers were less likely to have up-to-date lists of private schools in their area. And so when they were drawing up their plans for trainings and sending out the invitations —

Rob Wiblin: They just forgot them.

Karen Levy: — it was just more likely that they would get missed. And so we built into the process a list-making exercise. So just putting a little bit of effort into improving the quality and the comprehensiveness of the school lists — at that time it was at the district level; administrative boundaries have since changed — but that really made a huge difference in increasing the impact.

Karen Levy: There were other things that we found that weren’t really effective, so we stopped doing them. Certain forms of community sensitization didn’t really seem to actually make it more likely that people were knowing what was going on. So we stopped doing those things.

Rob Wiblin: There weren’t that many people with objections to this. Most people were just happy to go along.

Karen Levy: Well, you still have to be careful about educating people about it. So here’s a mistake that I made, where you say the wrong thing and you can really have negative effects. So in the very first year, we were physically helping to box up the deworming drugs at the national level. And we’re putting them in cartons, and I’m sitting there with a Sharpie writing “pills” on all of these boxes. And someone comes in and says, “What are you doing?” I said, “I’m writing ‘pills.'” He said, “You can’t call these pills — pills in this context often means contraception.”

Rob Wiblin: Oh, right. OK.

Karen Levy: The right word is “tablets.” So small things like that can make a really, really big difference. And community perception matters a lot. People can be skeptical of being handed drugs by the government: “Are they trying to do something to us?”

Rob Wiblin: You’ve used this expression of “mass drug administration” a couple of times. And I think if you were doing mass drug administration at UK schools, one probably would come up with a different expression for that.

Karen Levy: Slightly different set of expectations.

Rob Wiblin: “What are you worried about? We’re just doing mass drug administration for children. It’s responsible public health policy.”

Karen Levy: That’s right. [laughs]

Rob Wiblin: Sorry. Let’s get on.

Karen Levy: Exactly. Exactly. So yeah, all kinds of things like that really need to be worked out. And also helping to manage procurement. There are certain kinds of expenditures and activities that governments are really good at doing, and then there are others that are just structurally more difficult, so I think that’s another important way that partners can help. Procurement is a really good example.

Karen Levy: You don’t want it to be too easy for governments to procure things, but that process can take a really long time. So we were able to come in and, for example, get a whole bunch of training manuals printed in a way that was much more nimble and flexible — and allowed for the fact that we were still figuring out exactly what needed to be in that training manual — long after any procurement deadline would’ve passed.

Karen Levy: Another thing that’s difficult for governments, and is a particular problem for a program like deworming, is it’s really hard for ministries to collaborate. This is a big challenge that governments face. The Ministry of Health and the Ministry of Education each have their own budgets: you can’t send money from one ministry to the other. So if the Ministry of Education in theory wanted the Ministry of Health to pay for something, they would have to send the money back up to the treasury, and it’d have to get reallocated back down. So there were lots of times where us supporting a particular activity just unlocked the ability for it to happen,

Rob Wiblin: Made it much easier.

Karen Levy: That’s right. That’s right.

Rob Wiblin: That’s really interesting. We talked about this issue of procurement being extremely slow in some countries with Maha Rehman, in the episode where they were studying that and finding it was just a massive problem. The government was often paying many multiples of the true price of something, because people were so scared that it would take them months or years to get paid, if ever.

Rob Wiblin: So basically you were just buying all of the stuff ahead of time, and then you would be like, “Well, the Kenyan government is good for it. They’ll pay us at some point, but we’re just going to buy it and run ahead.”

Karen Levy: We bought it for the program.

Rob Wiblin: I see. Right. Right.

Karen Levy: And that was part of what we provided to the program.

Rob Wiblin: Yeah. Right. I see.

Karen Levy: And it was better for the government to use the funding that they had for —

Rob Wiblin: Things that you couldn’t buy.

Karen Levy: — things they were able to spend money on. Right.

Rob Wiblin: Just winding back a second. So you came up with the proposal for how things could potentially run, how this program could go, that people were generally supportive of. But did you have to do any work to persuade specific people within the ministries to make this a priority and to allocate staff to making it happen?

Karen Levy: Absolutely. And we did a lot of work at the technical level, at the Permanent Secretary administer level, to generate and maintain support for this program. We invited people to do deworming. We created media opportunities for them, so that they would get the credit that they deserved for supporting a program like this. Helping partners in government that you’re working with look good to their superiors and helping them achieve their goals is a really effective and important way to help unlock energy around these things.

Rob Wiblin: Action.

Karen Levy: Yeah, absolutely. And again, no one’s saying deworming is a bad thing. These are people who are working really hard, and getting pulled in a million directions, and have a lot of priorities, and are really underfunded — and so helping to make it easier to get these things to happen is just really important.

Rob Wiblin: What were some of the other motivations for your collaborators in government? So one is to get good PR to do a good job and have people see that you’re doing a good job. Were there other reasons?

Karen Levy: One of the most effective and important achievements was getting deworming coverage numbers included in the performance contracts of the Permanent Secretaries. Policies and priorities are set at the Cabinet level, and then flow down through line ministries and all of the technical people and the implementers in those ministries. And so if part of a Minister’s or a Permanent Secretary’s definition of job success is to have deworming coverage reach a certain level, that’s a really great way to align people’s focus.

Karen Levy: But the other thing that I think was also really important in Kenya is that Kenya became a model for other countries in how to do school-based deworming. And I think that was a justifiable source of pride, and it was exciting to see something happening at such a large scale. I think people really relate to the problem of worms from their own life experience, and that of their families and their children. So again, I think having partners that were willing to sit next to them and look at the problem together and see it through to the end really went a long way.

Rob Wiblin: A common complaint with people who work with governments, in this context and in all contexts, is a really rapid staff turnover — where a program could be turned upside down because the Minister changes. Is that a problem that you ran into?

Karen Levy: Yes. And one of the things that I have learned — and that I work with partners on now — is how do you codify and institutionalize relationships, such that they survive turnover? We all know how important it is to have champions in government, but champions leave. And so you need to find new champions. So relationship building is essential and never ends.

Karen Levy: But you also have to be thinking from the beginning about how to codify these relationships in a way that is not just an individual thing. So that can be MOUs. It can be membership on existing committees or technical working groups, getting written into annual operating plans, seconding people to ministries so that you have people actually sitting there in the office. These are all ways that relationships can be deeper and longer term and more enduring than you would get by just getting one individual person excited — which is a great way to start, but that needs to then lead to some of these other longer-lasting approaches.

Rob Wiblin: Did you ever have to coordinate with people who were in some sense unqualified or maybe out of their depth? And so perhaps weren’t up to what you were hoping they would do?

Karen Levy: Well, there were certainly times where I was out of my depth, let me tell you. I think that there is a perception of civil servants or working with governments that it’s slow, nobody cares. I actually have not found that to be the case. I find that by and large, the people that I have met and have worked with in the Kenyan government and in other governments are public servants: they are incredibly dedicated and they are working within really tough constraints.

Karen Levy: Sure, there’s gaps in capacity that we all have to face and work on together. But I really think that very often those perceptions come from misunderstandings about how decisions are made — who has authority to make certain decisions and why. So we can very easily say, “This person’s not being helpful” or “They’re being obstructionist” — but they’re working within a set of rules that is not of their making. So I think that there are way more opportunities to work constructively and collaboratively with government partners than we allow ourselves, perhaps because of some of those assumptions.

Rob Wiblin: Are there any other lessons that you learned about coordinating with governments and partners in government that you haven’t had a chance to mention yet?

Karen Levy: I think it’s really important for people to learn more about how governments spend money, how they make decisions, and how they deliver goods and services. I often like to use the university as an example when I talk to people about how you should be thinking about working with the government. So let’s say you wanted to do a project with a university, and you said, “Well, I’m going to go and I’m going to talk to people. I want to talk to a broad selection of people at the university to find out about this project. And I’m going to do that in July.” Well, anyone who’s been to a university, certainly in the US, would say, “But you can’t; nobody’s there in July.”

Karen Levy: Similarly in governments, there are fiscal years. Talking to somebody right at the end of a fiscal year is not the right time to be thinking about how they’re spending money. The plans for the next year have already been made. The money from the previous year has already been spent. So being cognizant of the cadence of the way that a government works is really important.

Karen Levy: Another analogy that I think works well with universities is, let’s say you wanted to say, “What does the university think about this?” People ask me that all the time: “What does the Kenyan government think about this?” Well, who do you mean? Shall we ask the president? Shall we convene the cabinet? Governments are these incredibly complex institutions, within which there’s a lot of disagreement. And at the same time there is policy that flows from the top. There are structures, there are annual operating plans, et cetera, that you need to understand how to work with.

Karen Levy: So I think that understanding the periodicity and cadence of the way that governments work is really important: budgets, election cycles, the time horizon for policy change. And meanwhile in the development sector, everything works to the timeframe of the donor project cycle: you need to put in your reports to the donor; they’re marching to their cadence. And so working with the government, it’s a big ship: it does not turn on a dime, and you wouldn’t want it to. So really recognizing that and embracing it as a pathway to massive scale that you have to work within and alongside is absolutely essential in terms of getting things done.

Rob Wiblin: Yeah. Are there any other lessons, while you’re on a roll here?

Karen Levy: Certainly some of the things I’ve mentioned about how governments can or cannot spend money, moving budgets from one line ministry to another — I had no idea about this until I started working with governments myself. I think that we could all do a lot more to ask the partners that we’re working with how the system that they’re working in works, and how best to support and work alongside that system.

Karen Levy: I also think one of the most important functions of a government is a coordinating function — and it’s all too rare to find development partners that are willing to be coordinated. I remember sitting in a meeting of a bunch of partners in the school health space. It was representatives from a bunch of different NGOs, all of whom had grants to coordinate the school health sector. Nobody was there to be coordinated. That’s what we should be letting governments do, and let them coordinate us. And that’s hard. That’s hard.

Rob Wiblin: Well, it’s giving up control. It means you can’t tell donors exactly what you’re going to deliver. Because someone else might tell you what they want to have delivered. That’s interesting.

Karen Levy: That’s right. That’s right.

Rob Wiblin: You were in charge of the country office in Kenya for five years, and I think you had some continued involvement with these projects after that. How long did it take for it to kind of mature?

Karen Levy: So we did the first national round of school-based deworming in I think 2008, and I think we reached 3.6 million kids. It was around 8,000 schools. That was the first sort of national-scope deworming program. The following year, the funds that had been available in the Kenyan government were no longer available. The donors had pulled the entire Kenya Education Sector Support Programme — of which the deworming program was a tiny, tiny little piece. There were much bigger issues and challenges, and the whole lot of funding got pulled. So the program basically stopped, and we set out to bring in new support for the program so that it could continue and continue to grow.

Karen Levy: We were finally able to get CIFF, the Children’s Investment Fund Foundation, to put serious resources behind it. They committed $12.8 million for five years of delivery. That started in I think 2011, and really that was what then created our ability to set up a machine that was going to just work, year after year. But it always needs to be tended — as I say, nothing is sustainable; nothing is self-sustaining without constant energy and attention and investment.

Karen Levy: And so thanks to GiveWell and other like-minded donors, the program continues. It has gotten more efficient over time. It leverages incredible investments of thousands of Kenyan teachers’ time and resources from the government, but does still require philanthropic support. In India, the government pays for a lot of the cost of the program, but support from philanthropic donors through Evidence Action’s Deworm the World Initiative plays a really important function in leveraging that government investment, and making sure the program is happening — and happening in an effective way.

Rob Wiblin: Having been around for a bunch of years, does the program become more resilient because there’s kind of pressure for it to continue, because more people are invested in it, parents expect it now, the teachers are planning to do it — such that if funding disappeared, people would create a ruckus?

Karen Levy: So I don’t think that deworming is salient enough for people to argue if it’s missing.

Rob Wiblin: To be sticky like that.

Karen Levy: I mean, when I was in school, we all got tested for eyeglasses. We all had our sight tested. If they suddenly stopped doing that, I’m not sure my parents would have noticed.

Rob Wiblin: Parents are too busy to notice.

Karen Levy: Yeah. And again, this is one of those things where it shouldn’t be incumbent upon people — they should just get this. So I don’t think that public pressure is the key to sustaining a program like this — it’s got to be something that is just very deeply embedded in annual operating procedures. It’s just what happens. It’s just something that happens every year.

Rob Wiblin: So the project reached a degree of stability between 2011 and 2016, thanks to this grant from CIFF, which was about $2 or $3 million a year. And that was mostly to pay for the tablets at that point?

Karen Levy: No, actually. Deworming drugs are contributed — they are free. There is a global drug donation program coordinated by the WHO in which GlaxoSmithKline, Johnson & Johnson, and other manufacturers donate deworming drugs. So to the programs, they are free. Helping to coordinate that was some of the work we did in the beginning. I think over time, there’s less to be done on that front.

Karen Levy: So no, actually the drugs themselves are a very small part of the cost. It’s really about the delivery. And so in Kenya, Deworm the World acted almost as a fiscal agent, supplying the funds needed down to the last-mile level in order to run the trainings, to do the data collection, to actually do the activities all the way down to the school level.

Rob Wiblin: To some degree we’re already talked about this, but a lot of people might have the reaction that the Kenyan government should be funding this, that it’s weird that a nonprofit is involved in coordinating this kind of thing. It seems like there was a test where that disappeared for a year and the program stopped — it wasn’t as if the government stepped up and paid for it. I guess there’s probably quite complicated, interesting, decision-making institutional reasons why it is the case that it’s not easy to grab a few million dollars and a bunch of staff to take over the role that you were doing. If you walk away, then the program freezes.

Karen Levy: Kenya is a resource-constrained place. So again, I mean, this may sound glib, but it’s kind of like saying, “Well, if poor people just spent more money on stuff, they would be less poor.” They’re poor because they’re poor. And so there are enormous pressures: there are lots of needs that resource allocators within the Kenyan government need to fulfill. Countries like Kenya borrow a lot of money. So I think it’s an overly simplistic mental model to say, “Well, there’s this big pile of money. The government, they’re so rich, why don’t they just spend money on this?” If you think about those funds as taxpayer funds: we’re convinced that the people receiving the deworming drugs shouldn’t pay for them, but that’s their government that we’re talking about.

Rob Wiblin: They would be paying for them through the taxes, effectively.

Karen Levy: Exactly. So, look, this can be taken to extremes. Of course there are things that governments can and should pay for. But in some ways, I almost feel like the amount of effort that it would take to ensure that these funds were allocated and dispersed and delivered and used well, year after year, is frankly —

Rob Wiblin: It’s easier just to pay for it.

Karen Levy: — we should do that for Kenya. We should just do that for Kenya. And there are other countries, like South Africa, that pay for it themselves. India is paying for most of it themselves. It’s leveraged by a much smaller amount of money that pays for the technical assistance and support around those basic costs.

Karen Levy: It’s a similar thing, like about blaming poor people for being poor. Kenya is resource constrained. There’s a lot of things they need to spread their resources over, and if this is something that can be easily taken care of by philanthropic funds, I would much rather see that happen. And hopefully there will be a time, not that far from now, where Kenya won’t need to have a deworming program.

Logistical issues [02:19:49]

Rob Wiblin: Makes sense. We’ve been talking here a bit about collaboration with governments or government-related issues. Let’s just talk a little bit more about the logistical issues that we left behind. You mentioned you found a bunch of other efficiencies, ways to run the program even better once it became a bit more stable, 2011 through 2016. What were some of those?

Karen Levy: Gosh, there were so many little details. When we first started, everything was on paper — digital data collection was very nascent. So we did a lot of work around designing better forms, so that they could be more easily rolled up and collated as they moved from the school level to the district and then the province and then the national level. Again, these were the early days — the program has now been run by teams for years since then. So a lot of this is old news now.

Karen Levy: There are new innovations and new technologies that are helping to do this more efficiently. Everything from how many days should the trainings be, so that you could fit in two in a week instead of one; this thing about making the lists so that you make sure that everybody is able to show up for the teacher training; optimizing the right number of tablets that go to each school so that you don’t have too many, but also so that you don’t run out. There’s nothing less efficient than having gone through all of the work to deliver a deworming program, then you run out of tablets on the day.

Karen Levy: So those kinds of things — again, the deeply unsexy but super important, and frankly, what I find thrilling. How do you get all of that to work, and how do you get it to work so that you don’t have to make decisions about this every time over and over again? That to me is really a sign of success: when it just kind of seems obvious, like it’s always been done this way.

Rob Wiblin: How did you get the tablets delivered to the schools on time?

Karen Levy: So in the early years, the first couple times we just sort of did it manually. But over time, we worked with existing drug distribution systems to get them to depots at the local level, and they would then be distributed to these teacher training sessions. So again, that’s another example of the need for coordination between ministries that in some ways an external partner is better placed to do than people within the government. It was a whole series of SOPs that we built for every step in that chain — moving things from the center, down a cascade, and then moving all of the excess tablets and the data back up to the center so that they could be processed there.

Rob Wiblin: So there are two classic challenges that come up in massively scaling any development interventions. You worry that the quality of the intervention is going to degrade as you reach way more people — so something worked well in one classroom, but now you’re trying to get thousands of teachers to do it. You haven’t trained them to do it properly, or they’re not quite up for it for some reason.

Rob Wiblin: Another one is what economists call “general equilibrium concerns,” where something might be fine just in isolation, but then as other people respond — as the entire system adapts to this new intervention — things could go awry, or there could be unintended positive or negative effects that you need to consider.

Rob Wiblin: Deworming seems like a pretty bad example of both of these, but I’m interested to probe you about what possible ways could the quality of the deworming get worse that you had to see?

Karen Levy: So you expect quality and fidelity to design to erode as a program grows, and so you have to build in those expectations ahead of time. You have to assume that there are going to be some mistakes in the estimates of how many kids there are. So again, you have to make a call: do you want to err on the side of sending too many or too few? I think that the best programs are robust to some fuzziness. If a program is cheap enough and if it is big enough, well, if it’s not the very best quality it could be, it kind of doesn’t matter, actually.

Rob Wiblin: It should be fine, yeah.

Karen Levy: You’re kind of picking all of the low-hanging fruit and that’s good. At the same time, if you are not aware of that dynamic and if you don’t build in mechanisms to shore it up, that will only get worse over time. So I think one of the most important factors is putting in place a monitoring system that will allow you to know what’s going on — not only because you can then respond to gaps and dips in performance, but also because the very existence of a monitoring program helps prevent an erosion in quality, because people know that there’s monitoring happening.

Karen Levy: So in the beginning you kind of monitor everything: you’re not sure what matters yet and so you’re measuring everything. Part of what happens over time is you start to identify those key things that, if we know that this thing is happening, we can be pretty confident that things are going well. So in the deworming program, for example, the presence of drugs at the school the day before deworming day is a really easy thing to monitor — it’s an excellent proxy for a whole bunch of other stuff having happened. You come up with things like that over time, that you can then look at and, in many cases, respond to in real time.

Karen Levy: And again, this is one of the functions that an external partner can really be incredibly valuable to a government. Management that can do this kind of stuff is hard to come by in general, but also, those people are super busy and they’re managing a lot of things — so complementing government programs with that kind of capacity can be really useful. I think you have to build that in from the beginning, and expect, “If 80% of this goes as expected, what do we think is going to happen?” If everything needs to go right in order for the program to work, the program will not work.

Rob Wiblin: What about the general equilibrium effects then?

Karen Levy: One general equilibrium effect that you might be worried about on a mass drug administration program, a deworming program, is drug resistance. If you’re regularly treating everybody over a very long period of time, will the drugs become less effective over time? This is certainly something that scientists pay attention to through surveillance. My understanding is that annual or biannual treatment is not enough to make that a serious problem, but that’s certainly something that you would want to watch out for.

Karen Levy: We worry more about general equilibrium effects when we’re affecting markets in ways. So if you are increasing agricultural productivity, are you then making the price of farm products go down or things like that? So general equilibrium effect questions are absolutely critical to be thinking about as you scale any kind of program, but beyond the drug resistance one, there aren’t major ones that come to mind for me with respect to deworming.

Rob Wiblin: Did you learn any valuable lessons about how to keep donors happy?

Karen Levy: I hope for a day when the way to keep donors happy is to deliver amazing impact for them. I think more and more we have donors like that, and I think that’s what’s so exciting to me: when the donors’ agenda and their definition of success is fully consistent with delivering impact.

Rob Wiblin: Is the same as yours.

Karen Levy: Sadly, there are other donors that are still out there that respond more to pretty pictures and good stories. And frankly, those are not the donors that I work with as much. What I think is most important, and what I have learned over the years, is about really being truthful about what you think you can achieve and what the risks are. It takes a certain amount of courage on both the grantee’s and the donor’s parts to have those conversations.

Karen Levy: Nobody wants to think about something failing, but things fail all the time. We’re trying to solve really hard problems. And the vast majority of private-sector businesses fail — and those are cases where everybody’s interests are aligned, and people are trying really hard to just make those businesses work, and even in the face of that, many of them just don’t. And here we are trying to solve these deeply intractable social problems of poverty and ill health and all of these — so we have to expect that a lot of these things are not going to work.

Karen Levy: I think that when we over-promise to donors, we do ourselves and them a disservice, because then there isn’t an environment where there is a tolerance for, and frankly, an embracing of failure — such that we can learn and then stop investing in those things that aren’t working well. So if you can get to that place and have that kind of relationship with a donor — even though in the short term that can be very scary — I find that in the long term, you can really get a much more trust-based, long-term relationship to support the work that you’re doing.

Rob Wiblin: So to bring this section to a close, how should the world feel about this deworming story on the whole? What’s the big picture?

Karen Levy: Well, there’s no question that deworming is one of the largest public health programs in the world. Evidence Action, who now leads the Deworm the World Initiative, has helped to deliver over 1.3 billion treatments since 2014 at a cost of less than 50 cents per treatment. There are just simply very few examples of this level of breadth and reach and scale at all. And so I think this is something to be really excited about and proud of.

Rob Wiblin: Yeah. It’s nice to see an intervention come from conception — or testing, I guess — people doing repeat testing to check and pin down exactly what the effect is. But also just this massive scaling, which is a coordinated effort between nonprofit folks, medical folks, philanthropists in rich countries, governments in the countries that are affected — it’s pulled in a whole lot of people. And as you’re pointing out, just looking at the number of doses delivered, it’s been at least a partial success, potentially a very big success.

Karen Levy: It certainly inspires me, that’s for sure.

The importance of a theory of change [02:30:54]

Rob Wiblin: Let’s push on from Deworm the World and deworming specifically, and think about lessons learned in your career in general, the ways things have changed.

Rob Wiblin: So you’ve been involved in scaling deworming, also this immigration program No Lean Season. Now, unfortunately, we probably won’t get to explore that much today. Have you been involved in scaling any other programs?

Karen Levy: Sure. There was a program called G-United in Kenya, which scaled for a while. Sadly, not continuing now, but we hope to revive that at some point. And I’ve been involved in an advisory capacity on scaling many programs in many parts of the world.

Rob Wiblin: What kind of generalized lessons have you learned about growing programs? Is there anything that’s kind of cross-cutting?

Karen Levy: The most important lesson that I’ve learned in taking a program from evidence to scale is the importance of starting with, and always coming back to, a theory of change. A theory of change is essentially a description of a journey — from where we are now to where we want to be — and a set of hypotheses about what needs to happen along the way to get us there. It’s a causal pathway from a set of inputs to a series of outputs, which then lead to a set of outcomes and then ultimately to impact.

Karen Levy: So for example, inputs and activities are like the program components. For a school feeding program, that would be like the food and the training and the funds. And that then leads to a series of outputs, which is, say, a certain number of children receiving meals at school.

Karen Levy: That then leads to outcomes, which are intermediate- or short-term changes that result from those outputs. In the case of a school feeding program, that might be more children are attending school and are able to concentrate better in class. And then that finally leads to your impacts or your goals, which are the long-term changes that result from this causal pathway. In our example: improved learning, better socioeconomic outcomes, et cetera.

Karen Levy: When you have a piece of evidence or a set of evidence that kind of maps that out, the key to scaling is to figure out how to automate and simplify and deliver at scale those initial inputs, such that that causal chain remains. And what is it that you need to test along the way in order to make sure that that’s still happening?

Rob Wiblin: So you want to make your theory of change quite explicit, and then see what are the parts that might not work as it gets larger. What are the things that we need to be paying close attention to, basically?

Karen Levy: Exactly. For different types of interventions, you’re going to end up having different weaknesses along that chain, right? So for example, with deworming, once you get tablets in mouths, you can be reasonably confident that those are going to kill the worms. That’s not something you need to relitigate. So coverage in that case — the number of children receiving deworming drugs — as long as you know that they’re not fake, that they are actual drugs and they are actually being taken, you don’t need to necessarily relitigate that all the time.

Karen Levy: However, in many other programs, particularly ones that have behavioral components, you have to double-check that. So for me, the biggest lesson is unpacking the evidence into a really robust theory of change, having a very clear strategy for delivering those inputs and those activities in such a way that you’re going to set off that causal pathway, and so that you’re going to know along the way if each successive step is happening.

Karen Levy: When we were talking about some of the examples in deworming like, “Are there tablets at the school? Are all the schools on the list?” — those are some of the assumptions that we had to question along the way, and make sure that that entire journey from inputs all the way to impact was really happening.

Ways that things have changed since 2006 [02:35:16]

Rob Wiblin: What’s a way that evidence-based development practice is different today than it was back in 2006? Important things that we’ve learned as an industry?

Karen Levy: Absolutely. First of all, we have an incredible infrastructure of organizations in this space. There are lots of people that now have technical capacity that didn’t before, that are available to work with implementers and governments. There are people within implementing organizations and governments that have done this kind of work. So there’s just more knowledge and experience and resources that can be wielded to help answer questions about what works, what doesn’t, et cetera.

Karen Levy: I also think that there’s been a big growth in the sophistication of how we think about generating evidence for policymakers, and what we mean when we say “moving from evidence to scale.” I think we used to have a slightly more simplistic view — and perhaps the deworming program is an example of that — where it’s a program, a package, you study that thing, we want to scale up that thing.

Karen Levy: Now there’s bodies of evidence that speak to each other in ways that are more than the sum of their parts. We can pull in lessons about what motivates frontline workers in Zambia, and use that to inform program design of some other subject in some other place. We’re also seeing that there are certain approaches that have been subjected to sufficient numbers of studies — like Teaching at the Right Level, where the evidence base is so strong that we really now can be thinking about how do we adapt this and deliver it in a range of places? And Teaching at the Right Level is now its own organization, doing that in incredible ways in multiple countries.

Karen Levy: So I just think that the whole field has gotten much more sophisticated and nuanced, and there are just way more opportunities for making evidence generation useful and salient to actual programs and policies.

Academic work vs policy work [02:37:42]

Rob Wiblin: What are some ways that the skills or the mentality of a researcher — someone who’s trying to publish paper — needs to be different than someone who’s trying to scale a program, someone who’s trying to deploy it, or someone who’s advising a government, or trying to get the government to do something useful?

Karen Levy: Rachel Glennerster has a wonderful talk on this topic that I really encourage everybody to listen to. She gave a keynote at CSAE a few years ago and talked about some of the differences in doing evidence generation as an academic versus for policy, and I think some of the themes she pulls out are sort of worth talking about.

Karen Levy: In academia, you’re trying to find the optimal answer or the optimal solution. But in a policy setting, you’re trying to optimize within constraints — you don’t have an unlimited set of options from which to choose. So the way you would construct a study or the way you would ask a certain set of questions is really very different. The timeframes are really different. What success looks like and how one is rewarded for a study happening well or not — they’re just very, very different.

Karen Levy: But even though these differences exist, many of the examples that we have of programs that have scaled successfully have been ones that have been driven by academics who really care about the policy influence of their work. I think the reason that matters is because evidence isn’t just a set of answers that you pull off the shelf. In taking something from a study to scale, all kinds of questions come up about, “What are the underlying mechanisms? Why did we do it that way? If we change that, are we now pulling apart the whole theory of change, or is that still going to work?” So I think some of the most exciting examples of times when studies have moved from evidence to scale have been when you have a really committed academic, who’s willing to kind of take that journey with policymakers.

Karen Levy: There’s a paper that Michael Kremer and Sasha Gallant and a few others did, looking back at investments that DIV (Development Innovation Ventures) made over a certain number of years. And this is one of the things that they point out, this relationship between academic involvement and scale.

Rob Wiblin: Are there any other lessons that you’ve learned about scaling from a paper to a massive program that you haven’t had a chance to share yet?

Karen Levy: Well, I have learned an awful lot about how to make those academic-practitioner partnerships work well, and so maybe that’s worth sharing.

Rob Wiblin: Yeah, go for it.

Karen Levy: There is sometimes some tension between what a program needs from a data and evidence perspective, and what is attractive to an academic. Sometimes you really just want to replicate something, or you need a bunch of descriptive statistics — and this is not necessarily what’s going to attract the attention of a world-class development economist. At the same time, many of those world-class development economists, we’re lucky to have them involved and caring about the impact of their research.

Karen Levy: So I find it’s really useful and important to have an open conversation with academic partners about what would make it interesting for them. I have found that almost always people are absolutely willing to do the less sexy stuff, if you also allow them to ask some really interesting, innovative questions that will drive their research agenda — and nine times out of 10, that part of it is going to end up leading to new insights that affect your work anyway. So the marginal cost of adding another module to a survey, or trying a slightly different version of an intervention — doing that can create the space for intellectual engagement from top thinkers, and then brings those people to the table for all of the other things that you need. And so I think we should have those conversations.

Rob Wiblin: It sounds like you think that people in government or people doing deployment can sometimes be too reluctant to add on things to their program that are basically just for the interests of these academics — even though on net, it might be a real win-win.

Karen Levy: I think we should be open to that when the cost is low. If it’s literally just adding some questions to a survey, it’s not even visible to people who are implementing. But it’s rare that you have a situation where everything is kind of asked and answered — there are always interesting and important questions to be asked. So I think a real, honest conversation on the part of donors, government partners, implementers, and academics about “What does it take to bring everybody to the table?” — I think that’s a conversation worth having.

Fit for Purpose [02:42:49]

Rob Wiblin: So we’ve been going for a while. Maybe it’s time to begin wrapping things up and bringing things to a close. As a last major topic, I’d be keen to talk about your current consultancy project — the thing that you’ve now gone into, which you mentioned at the start — called Fit for Purpose. What are the main challenges with the aid industry that Fit for Purpose is designed to take on?

Karen Levy: So we talked earlier about some of the ways that the aid industry is structured, and how this can kind of create misaligned incentives. That is because most of the players in the aid industry are either donors or grant recipients. I’ve spent most of my career working with and helping to build organizations that are aiming to achieve impact. And these organizations I believe are really important — we need them to exist, and I feel very good about my role in helping to create them and am cheering them on.

Karen Levy: At the same time, there is a point at which the organization always becomes the project. And I understand why this is the case: as the leader of an organization, you want that organization to continue. For me, it felt like I was ready to almost indulge myself in having some time to work in the industry, such that the work that we are doing is fully aligned with impact — where the project is the impact. For us, a big impetus behind starting Fit for Purpose was to create a mechanism to engage in and to contribute to global health and development in a way that is aligned with impact.

Karen Levy: So our business model, as a small firm, relies on really adding value to our partners and our clients. Our partners and clients pay us for our time and for the value of the work we contribute — so I don’t have skin in the game about who gets funded or who’s getting the overhead on a particular grant. I’m really loving being able to work with and provide strategic and technical support to some of the world’s best and most innovative social entrepreneurs and researchers and philanthropists. And I’m finding it really rewarding to be able to use my expertise, my networks, and skills to help them turn up the volume of their work — in a way that I can be really honest and open about what I think makes sense for impact.

Rob Wiblin: What sort of questions do people come to you with?

Karen Levy: There’s a big range. We have a portfolio now of about a dozen partners that we’re working with. I can use GiveWell as an example — they are a partner that we’ve worked with since the very beginning. And I should say, I am not a GiveWell employee. I’m not representing them, but I have been providing ongoing strategic and technical support to them and their team over the last two years.

Karen Levy: And it’s a range of types of work. In some ways, providing the perspective of an implementer as they think through their research. Some insights on supporting RCTs, and ways to make that effective and maximize the impact of those RCTs. Providing some context on delivery- and partnership-related issues.

Karen Levy: But one of the aspects of it that I’m finding most exciting and rewarding is serving as sort of a bridge or a translator for organizations who are potential grantees, who are less familiar with the GiveWell approach — because so much of what they do is so different from the typical way that the aid industry works. A lot of the muscle memory that grant seekers have needs to be unlearned a little bit — in ways that I think are refreshing once you get there, but are not necessarily obvious.

Rob Wiblin: What’s an example of that, where people have the wrong intuitions about what to do or say?

Karen Levy: So for example, this notion of sustainability: “Don’t worry. We’re going to just invest a little bit now and then it’s going to take off on its own.” That’s not necessarily consistent with really understanding what is the most cost-effective way that we can invest in this program now and potentially over time. That’s a very clear example. Another one is thinking about the fully-loaded costs of an intervention. Many donors want their funding to leverage that of others. Now, that can be a good thing. However —

Rob Wiblin: Hard to generalize.

Karen Levy: — just the mere fact that someone else is spending a lot of money on it doesn’t necessarily make it a worthwhile investment. Of course GiveWell thinks about the total costs — whether they’re borne by a GiveWell-inspired donor or some other source of funds — thinking about that whole package.

Karen Levy: Another important aspect is thinking about the fact that not every intervention is going to have the same level of cost effectiveness in every context. Different countries have, for example, different disease burdens. So really understanding that and making a prioritization matrix, for example, of where you might want to deliver a program. That’s a very different approach from some other donors, which say, “We fund these countries.”

Karen Levy: So some of these ways of thinking — once you actually talk to people and can get them to let their guard down from what’s the typical donor face — I think most actually find it a really refreshing relief. But you need to hear that. We have found that it can be helpful to have someone like me who’s been both a grantee and now an advisor to GiveWell to talk through that process. So that’s been something that’s been very exciting.

Rob Wiblin:
That’s really interesting that you have one foot and a bunch of history in the traditional nonprofit development world, and also one foot and a bunch of experience with the effective altruist mindset and effective altruist research style — and I guess the way that we talk about things as well. So you can potentially be this translator and explain the concepts that people have back and forth. A lot gets lost when people are talking to one another, because in some cases they’re not being 100% frank or candid about everything that they think. In other cases, they just assume that people have knowledge that they don’t.

Karen Levy: Yeah. As we’ve been saying, frankly, a lot of the aid industry is often based on these weird perverse incentives. When you really strip those away, we have to talk to each other differently about, “What are the real costs? What are the real benefits? And what is the uncertainty?” Most donors don’t want to hear the likelihood of failure, or they want that number to be zero. And again, I understand that, but the reality is there’s risks in everything we do. A lot of the things we try may fail. And forward-thinking donors like GiveWell and now others as well are starting to kind of bake that into their assessment of grants, and are more and more willing to take some risks — understanding that not everything is going to work out in the end.

Rob Wiblin: Foundations are famously very diverse in what they want, but I imagine that if you went to most of them and were very upfront about all of the weaknesses of your program and all the ways it might fail and all the concerns that you have about the project, they probably wouldn’t fund you because they’d think, “Well, if this is what they’re telling us, imagine what they’re not telling us.”

Karen Levy: That’s right.

Rob Wiblin: Going to GiveWell and smoothing things over is a big mistake, because then they’d be like, “Well, what aren’t they telling us?” And if you’re upfront about all of the things you understand that could go wrong or the weaknesses in the evidence, they’re going to think, “Wow, these guys are really smart. They’re switched on. They’re being honest.” And so it’s quite the reverse of the strategy that you would normally take.

Karen Levy: Precisely. So if you have the luxury of being able to be truly impact focused, then you have to embrace that uncertainty and be thinking about, “What are all of the things that can go wrong? What are the likelihoods of those things going wrong?” And also, “What are previous failures that we’ve learned from?” This is not a way to get new funding from most donors: to tell them all the things you tried that failed.

Karen Levy: And as we’ve discussed, in the private sector, things fail all the time. There’s very little tolerance for that in the nonprofit sector, and yet here we are trying to solve these really tough problems. So sometimes it just helps to hear from an honest broker, like an earnest outsider that says, “It’s OK. You can say that, and that is actually a credibility-building conversation in this context.” And for the right kind of organization, that’s really refreshing.

Rob Wiblin: With the first question about what Fit for Purpose is up to, there’s all of these potentially perverse incentives for people in the development industry, and I guess you don’t feel those so much running your own consultancy. Can you elaborate on that? I’m not exactly sure what you mean and what’s different.

Karen Levy: If you’re working at an organization that is a grant-seeking organization, as we’ve discussed, it’s highly likely that your business model revolves around the quantity of funding that you’re able to raise. And so, if you can raise $20 million and your overhead rate is X percent — again, whether X is high or low — that is what then enables you to pay your finance team, pay your office rent, and pay the electricity bills. Well, if along the way it turns out that actually, it would be much more efficient if this other organization was delivering that — or let’s say the government was going to be able to pay for some of it. Well, in a funny way, your incentive is not to let that happen, because then you lose your X percent on that amount of money.

Karen Levy: So as a partner — as Fit for Purpose — we can truly be an intellectually independent, honest broker and talk to a set of partners about what makes sense for the impact here.

Karen Levy: When I did Y Combinator, I expected it to be a really competitive environment: here you are in the private sector and it’s all about competition. And I was blown away by the level of collaboration that existed in that community — and frankly, in comparison to the nonprofit world, which can be competitive. People compete for funding, and so very often we’re fighting over slices of the same pie. Whereas the Y Combinator model is like, “We’re making the pie bigger. It’s getting bigger for everybody.”

Karen Levy: So again, as an honest broker, independent voice, and partner, I aspire to help bring actors together in ways that we can become more than the sum of our parts — where there isn’t this kind of zero-sum game, like, “You’re going to get the grant or I’m going to get the grant.” I think that there’s a lot of work to be done and a lot that we can explore as an industry on how to collaborate and work together more effectively.

Rob Wiblin: What’s the thorniest problem or hardest question that someone has brought you so far? You might not be able to say, but if you can.

Karen Levy: Coming back to the sustainability question is always a tough one. Sometimes it’s helping people to articulate to their donors why that should or shouldn’t matter, and helping people think through how to communicate that with the people that fund them.

Karen Levy: Another related one is about diversifying funding: many donors see a diversification of funding sources as, in and of itself, a good thing. And that may not be the case, in the sense that if you take an effective altruist approach — you know, philanthropic funds are philanthropic funds — and having a diverse portfolio of grantors often means you just have that many more donor reports to write. And the restrictions that donors can place on how you spend their money ends up making for lots of inefficiencies. So that can be a tough conversation.

Karen Levy: But I would say challenges in working with governments, where you’re getting government pushback on something, and how to work through those relationships. And management issues — I’ve kind of seen it all from one point of view or another. What I really like is being able to be a shoulder to cry on or a wall to bounce ideas off of. So leaders of organizations, or programs within organizations, can come to us and get our help in working through these thorny problems in a way that doesn’t feel high stakes — like, “My donors are going to abandon me because I don’t know how to sort this out.”

Rob Wiblin: How do you enjoy consultancy work on a day-to-day basis, compared to the work you were doing before — actually trying to scale one specific program, and it’s the same program year after year?

Karen Levy: I’m still learning. This is still a new journey for me. One of the things I was worried about when I started about two and a half years ago was, would it feel isolating? And then of course the entire world got isolated. And so, we’re going to hold that constant — this is sort of a COVID fixed effect, and I’m just not sure how that’s going to feel moving forward. I was certainly worried about whether or not I would feel part of a professional community, and that has not borne out at all. I’ve just been very lucky to now be part of several communities in ways that I feel are really rewarding and exciting.

Karen Levy: It is still a little awkward for me. I’m still not used to asking people to pay me for my time, as someone who had been salaried. Even though I was always in positions where I was responsible for raising funds for my team and my salary, and so always felt a sense of ownership, it felt a little less transactional on a day-to-day basis. But again, I think for me, it forces me and the partners I work with to think about, over and over again, if I’m adding value and if it’s worth it. I feel like that’s how it should be.

Karen Levy: So, we’re still learning. We’re still figuring it out. But so far I have really enjoyed the ability to curate what I work on — to choose to take on a project because I think it seems really promising. And I’m just very fortunate to be able to take a little bit of risk at a personal level and see if this works out financially in the long run. So, so far so good. But we’ll see and watch this space.

Rob Wiblin: Is there anything else you want to say on the GiveWell or EA involvement stuff?

Karen Levy: This is related to something we discussed earlier. As GiveWell is now looking to expand the number and scope of programs that they are supporting, I think they are looking to engage with new partners, and that’s been very exciting for me to participate in. They are also hiring. It is a very exciting time to be joining their team. So, for your listeners who are interested in contributing to that, you should absolutely check out the GiveWell site and look at some of the jobs that they have open.

Rob Wiblin: Is Fit for Purpose hiring at all? Do you have expansion plans?

Karen Levy: We’re not hiring right now, but we are certainly always looking for like-minded people who may want to engage with and through us to contribute to the sets of partners that we’re working with now. And I’m certainly interested in hearing from and potentially engaging with social entrepreneurs — both starting their own organizations or potentially starting programs within organizations — that might be interested in engaging with us, so that we can help them.

Living in Kenya [02:59:41]

Rob Wiblin: Something that’s a little unusual about you — actually, I’m not sure how unusual it is in the development world — but you moved to Kenya 20 years ago, and you’ve basically been living there solidly for 20 years in the location that you were originally doing the work on deworming for quite a long time. What do you enjoy about living there?

Karen Levy: I remember when I first came to Kenya — it was actually in the ’90s — I decided not to put a cap on how long I was going to spend there. People used to ask me, “How long are you planning on spending in Kenya?” And I remember making a very distinct decision not to answer that question, because I felt like if I did that and if I said, “Well, I’m going to be here for two years,” the second you do that, it’s like now the clock is ticking — and every day that goes by, your level of investment and the amount of time ahead of you that you’re spending in a place is getting smaller and smaller.

Karen Levy: And so I may have overdone it. Here I am all of these years later. But I really do think that that changed the way that I formed relationships with people and with the place. What is a little different about my experience living in Kenya is, I think there are lots of people that work in global health and development that spend long chunks of time overseas from where they were originally born — it’s a little more common to hop around and spend a few years in each of several places. And I wouldn’t say that there’s a better or worse version of this, but I will say for myself, I feel like the layers of learning just get deeper and deeper.

Karen Levy: I remember when I had been in Kenya for six weeks, thinking I was so clever because I had been there for so much longer than the typical tourist. And then in six months I felt like, “Oh, gosh.” And I felt that way after six years, and I still feel that way today. So I feel like the learning never stops, and as long as I feel that way, I will be happy here.

Karen Levy: I also just can’t help but feel that the incredible diversity of people, of ecosystems, and of livelihoods that Kenya represents is just thrilling to me. And the sense that the future will be better than the present. I am driven by and really passionate about good, hard problems — but also the sense that we can make progress against them. And I feel that in Kenya in a way that’s very compelling. I’m not naive: there are serious challenges. I also live a very privileged life there. But there is this sense of motion and dynamism that I find electric. And if anyone wants to come visit me in Nairobi, I’ll show them why.

Rob Wiblin: Kenya is a country with plenty of poverty. But I suppose it might have a bunch more optimism about the future than the United States does, because maybe starting from a lower base, it’s got a long way up to go that might be relatively straightforward, potentially.

Karen Levy: The changes that I have seen in my own time here… I first went to Kenya in 1992. Kenya was a single-party state. There were sedition laws on the books — you couldn’t criticize the government. There was very little infrastructure. Today, I can do more with my mobile phone in Nairobi than you can in the UK, or that anyone in the US can. There are aspects of life in Kenya that are super cutting edge and innovative. And sure, there are still major challenges in poverty, but my day-to-day experience in Kenya is one of ultimate optimism.

Rob Wiblin: I suppose the ultimate conclusion of this — never putting an end on things — would be becoming a Kenyan citizen, and potentially retiring there and just living out the rest of your life there. What do you think are the chances you might do that?

Karen Levy: So, did my mom put you up to that question?

Rob Wiblin: Always. Is that something that comes from your family?

Karen Levy: I’m just kidding. I would never say never, just in the same way I used to answer the question of, “How long are you staying?” I would say, “Not permanently but indefinitely.” And now my life is here, my friends are here, my personal and professional communities are here. So there’s nothing pulling me away from Kenya. If there was, I might entertain the call. And I have certainly loved, in the last 10 years or so, expanding my professional experience to other countries in the region, to some new places in South Asia, where my learning curve is much steeper now. That has been a really nice complement to my base in Kenya. I feel quite committed to staying.

Underrated life advice [03:04:38]

Rob Wiblin: Cool. I guess we should wrap up, but maybe a final question (and possibly it could turn into a couple of related questions). You’ve been around; you’ve experienced a bunch of things. Is there any underrated or valuable life advice that you like to tell people that I haven’t yet given you the chance to share?

Karen Levy: Well, maybe unsurprisingly — based on what I just said about my experiences in Kenya — I would really advise people, particularly those who are interested in global health and development, to spend a substantial amount of time living somewhere other than where you grew up. And this is not only because you learn about new places, as valuable as that is. Really, it’s because of what you learn about yourself and where you come from.

Karen Levy: We expect to feel a sense of culture shock when we visit somewhere new. But when you spend long enough far from home, you get to experience that feeling about your own home place and culture, and see it with new eyes. And it helps you understand how much is socially constructed — it helps you see the illogic or the internal contradictions of where you come from. I think it actually really helps to develop this scout mindset that Julia Galef talks about: the ability to observe and question what you take for granted as true or obvious or normal.

Karen Levy: In this line of work in particular, I think it’s really critical for us to approach other cultures and institutions with the presumption that they probably have some of their own internal logic, and it’s incumbent upon us to try and understand it. And that conversely, we have lots of views and assumptions that are not necessarily logical or true, and that we should really be willing to question them.

Rob Wiblin: Yeah. Anything else?

Karen Levy: Well, I will just end by sharing one of my favorite stories about this very topic. Some of the work I did in my earliest days in Kenya was running study abroad programs for American high school and college students in Kenya. And not surprisingly, they — as I was when I first came to Kenya — were always very interested in the exotic. That’s what attracts us. So I would often get questions, especially if we were visiting a rural area, about tribal dances or ethnic practices or things like this.

Karen Levy: One of my very favorite things to do with groups of students in talking about cultural practices and “tribal traditions” is, I would say, “Everybody has tribal dances.” And they would say, “Well, everybody but us.” It’s the equivalent of, “Everyone else has an accent and I speak normally.” So I would say, “No, no, no. Absolutely everyone has tribal dances.” And everyone would be very suspect of this. And so I’d say, “OK. Everybody get up and stand in a circle.” And everybody would get up and stand in a circle. And then I would say, “You put your right hand in, you put your right hand out.” And of course, everybody knows the Hokey Pokey — and what is the Hokey Pokey, if not an American tribal dance?

Karen Levy: This was always one of my favorite examples to get people to think about this very issue that we’re talking about. We all have culture. We all have bias. We all are exotic. We’re all irrational. And being able to see your own place through those eyes really helps us develop a sense of inquisitiveness and truth seeking about the world that I think is very valuable.

Rob Wiblin: My guest today has been Karen Levy. Thanks so much for coming on The 80,000 Hours Podcast, Karen.

Karen Levy: Thank you so much for having me.

Rob’s outro [03:08:27]

Rob Wiblin: Before we go, I wanted to let you know about a free service some of you might be interested in. They’re called “Virtual Programs” and they’re not run by us, but rather effectivealtruism.org.

They’re basically short online courses that provide the opportunity for anyone around the world to engage intensively with the ideas of effective altruism through books, videos, podcasts, exercises, and weekly small-group discussions.

Each program lasts eight weeks, and you’d have weekly one-hour chats with a group of 3–5 participants and one facilitator. Before attending each discussion, you’d spend roughly two hours completing a set of readings and sometimes a brief exercise.

There’s currently three different courses: The Precipice Reading Group, Introductory EA Program, and In-Depth EA Program.

New groups start each month, and the next round of programs will run from April 4 to May 29, with applications closing March 27.

If you enjoy this show and would like to actually talk seriously with other people around the world who share your interests, then these virtual programs might just fill a gap in your life.

You can find out more at effectivealtruism.org/virtual-programs

All right, The 80,000 Hours Podcast is produced and edited by Keiran Harris.

Audio mastering and technical editing by Ben Cordell.

Full transcripts and an extensive collection of links to learn more are available on our site and put together by Katy Moore.

Thanks for joining, talk to you again soon.