Transcript

Robert Wiblin: Hi listeners, this is the 80,000 Hours Podcast, the show about the world’s most pressing problems and how you can use your career to solve them. I’m Rob Wiblin, Director of Research at 80,000 Hours.

Today I’m speak with a data scientist at UC Berkeley’s Centre for Effective Global Action.

If you’re interested in maths, social science, academia or any work focussed on helping developing countries, this is one you shouldn’t miss.

If you’re somehow listening to this and not subscribed on your phone yet, get a podcast app and throw it on there. That way you can speed us up to the ideal level, listen whenever your brain would otherwise be wasted, and never miss an episode.

If you’ve got room for other podcasts to listen to, can I suggest trying BBC4’s More or Less. It’s a guide to numbers in the news hosted by economist Tim Harford. If you enjoy chasing down statistics being cited all over the place to check whether they’re bullshit or not, it will be your kind of show – I listen to it every week.

And now I bring you, Ofir Reich.

Robert Wiblin: Today, I’m at EA Global San Francisco, speaking with Ofir Reich. That’s correct, right?

Ofir Reich: That’s pretty good. Ofir Reich is the full on for advanced version, but that’s good.

Robert Wiblin: I’ll practice later. Ofir joined the Center for Effective Global Action in 2015 as a data scientist, where he works on several data intensive research projects around global poverty. He was previously a mathematical research team leader in a technology unit of the Israeli army, and then chief data scientist and machine learning expert for a Tel Aviv based startup. He holds a mathematics and physics bachelor of science from the Hebrew University in Jerusalem and now has eight years of experience in groundbreaking applied mathematical research. Thanks for coming on the show, Ofir.

Ofir Reich: Thanks, Robert. It’s great to be here. I’m happy to take part.

Robert Wiblin: So in the second half of the interview, I hope we’ll get to talking about how people can actually solve global poverty using their quantitative skills, but first, tell us a bit about the Center for Global Action at Berkeley and what you do there.

Ofir Reich: Sure. So the Center for Effective Global Action or CEGA, C-E-G-A is the acronym, is based in UC Berkeley. It has three main activities, so I’m going to separate those out even though they’re all sort of under the umbrella organization. It was founded, I think, seven or eight years ago by Professor Ted Miguel, who’s a very renowned development economist. So these three separate parts are, one, the global networks where CEGA brings in scholars from developing countries to spend a semester at UC Berkeley, forming ties with other academics, learning, being exposed to more impact evaluation work and then going back to their countries and hopefully leading a prosperous academic career and also having these partnerships. Hopefully, creating impact through both research and through relationships with governments in their countries of origin. So that’s one, that’s global networks.

Then, the second activity is called the Berkeley Initiative for Transparency in the Social Sciences, or BITSS, with two Ss and that’s basically trying to address problems in the social sciences known as the reproducibility crisis. So many of the results reported in science journals are either false, or problematic, or don’t reproduce, meaning somebody tries to recreate the same research and then they get different results and so that’s a big problem for science.

So the Berkeley Initiative for Transparency in the Social Sciences, along with some other actors in this field, tries to promote better science practices, such as open data, where you have your data … if you do research, your data should be available to other people to try and see if you maybe analyzed it wrong or if there’s some other reason and trying to build on your work. And you should also be publishing your codes so that your research is reproducible and people can see that you just don’t simply have a bug, for example, or they can see what you’ve done, exactly what observations you drop and all these practices that then could be used, even sometimes maliciously, but most of the times not maliciously, to generate results that later turn out to be false.

And then preregistration, which is pre-committing to what analysis you’re going to do to prevent what’s called p-hacking where you just try a bunch of different tests until you get something that’s significant and publishable and then you publish that. Obviously, leading to problems later on. So that’s trying to help social science be better and more reproducible and more open. So that’s the Berkeley Initiative for Transparency in the Social Sciences, or BITSS.

And there’s the third branch of CEGA, which is about global poverty, specifically, and about academic research about global poverty. So this is a network of academics. Over 70 academics that are affiliated with CEGA and also are at various universities on the west coast of the United States and Canada, that all do research and development economics or a related field but all related to global poverty. So some engineers and some computer scientists, all trying to invent and then do research and test solutions to problems of global poverty.

Robert Wiblin: And that’s the one that you’re in?

Ofir Reich: And that’s the one that I’m in. Yes. So CEGA, the staff, directs these donor funding and competes it out to academics to produce top-notch academic research about the problems of poverty so that we understand it better, find better tactics for poverty alleviation and so on.

Robert Wiblin: So what kind of specific data science research projects are you working on at the moment?

Ofir Reich: So I’m currently working on three projects. One is a project about value-added tax evasion in a state in India, where we are trying to better target companies that are set up in order to evade taxes and thereby increase government revenues. Because if you can find these companies that are false companies just being set up in order to create this false paper trail, that they just then use to avoid paying taxes and basically defraud the state, then if you can target these better and find the ones that are fraudulent, you can increase tax revenue, which is good because then it will be spent on the poor in the state.

And so we are using data of the past years of tax returns, value-added tax returns, in conjunction with companies that are known to have been found to be these false companies to create a targeting mechanism using machine learning basically. So reviewing these tax returns and the companies we know to have been found in the past to be fraudulent and thereby automatically finding a way to target companies that are more likely to be fraudulent and then target them for inspections.

Robert Wiblin: So it’s a little bit, what, like the tax office might do here or what Visa and MasterCard … the kind of machine learning that they’ll do to try to identify fraudulent transactions. Is it similar work?

Ofir Reich: I think it’s similar in some respects. Yes. I mean, we have different data than some of those have and everything is very different in the developing world so a lot of things are physical. A lot of the data integrity is an issue because we don’t generate our own data, but, yeah, there are similar lines. And I actually worked in fraud detection before. That’s my background.

Robert Wiblin: In the private sector?

Ofir Reich: In the private sector when I first started up back in Israel.

Robert Wiblin: What are some of the other projects?

Ofir Reich: So a second project is working with an organization called Precision Agriculture for Development. This organization tries to improve agricultural extension, using information communication technology or basically mobile phones in developing countries, right, so we know that often, farmers don’t adopt the best practices in developing countries through research and we think some of this adoption can improve by having agricultural extension be more effective.

So for example, if you inform people about the benefits of fertilizer and advise them how to use it and how much they should use right before the time that they are about to buy fertilizer, they might buy the right amount of fertilizer and then use it on their field and then increase their yield, and more importantly, their profits. So the basic assumption is that even if the effects that you get are not enormous because not everybody’s going to listen to a message on a mobile phone, still the cost is so little that in cost effectiveness terms, this can be very large.

So we also attempt to bring in other factors into this such as weather and prices and all these to create, what we say, precision agriculture and soil test maybe. So the organization works on these kind of large scale projects. So I’m involved. There are a few of these around the world. I was involved in one in India, in Gujarat and then, now, starting to get involved in one in Ethiopia with the government of Ethiopia.

Robert Wiblin: So the data science aspect there, is that you’re trying to figure out what actual advice you should offer the farmers, given current weather conditions and soil conditions and so on?

Ofir Reich: Not necessarily. So there’s a bunch of things. The first thing you could do is trying to … I mean, they’re a lot of data problems that are related to this. So one of them could be, yes, we have all these data sources for weather and we want to know how reliable they are, how to use them and produce a recommendation, but also, you want to know things about your users, right, like what works better for our users. Does it work better if divide the menus this way or that way? How to see if people are not interacting with your system in the way that you would want. So, in a sense, this is more like standard research conducted in many companies that have —

Robert Wiblin: Like product engineering?

Ofir Reich: Yeah, but then you apply it in this way that’s not necessarily standard. But then there are various other data work as well. There’s working with soil tests and realizing how to interpolate them in the right way in order to give a recommendation on each location, not just where the soil tests were taken, how reliable those are, and many others.

Robert Wiblin: Do you ever worry that you might be sending out bad advice or are you pretty confident that it’s better than what people are doing already?

Ofir Reich: So we often rely on other agencies to produce some of this advice. So, for example, you have the agricultural authorities in several of these countries that do rigorous testing and have test plots and have a lot of research around this and we also try to very rigorously test what we’re doing. So we’re hoping to constantly improve this and also, always note and always measure.

Robert Wiblin: How do you test it?

Ofir Reich: So we often have either A/B testing for many things that you can find with your systems. So if you’re familiar with that term, it means, basically, you have a group of users and then for half of them you randomly let them use the services that exist today and for the other randomly selected half, you let them use the variation of your service and then you compare the interaction of those two groups with your system, and you see how it works.

But we also do many randomized controlled trials or RCTs for short. So these would be having some kind of change to the system, and then measuring outcomes either in calling people and asking about their satisfaction, about their change of behavior and so on or, ideally, really measuring yields. The problem with this in agriculture is that it takes a long time until you actually see the yields and so you’re searching for things that are a little more immediate.

Robert Wiblin: Yeah. I was thinking, you’re going to send texts to some people and not to others and then you’d have to go out and measure the yields or, I guess, ask them to say what yields they got. Is that how you test it or-

Ofir Reich: So that would be testing the yields in the end, but yeah, for example, there was a randomized controlled trial that sort of spawned Precision Agriculture for Development. This organization, which is in India by Shawn Cole, one of the co-founders of the organization and Nilesh Fernando, where they did just that. They had a voice service where they would call farmers once a week with an automated message about relevant practices for that week. Farmers could also call in and ask questions and those would be answered by agricultural experts.

So they did a randomized controlled trial where some people were exposed to their service and some weren’t. They then asked them for their yields and they found very good returns to the investment in the order of ten times the monetary investment.

Robert Wiblin: Is there a third project?

Ofir Reich: Yes. So the third project is … I’m not going to go into all the details, but it’s around mobile salary payments for teachers in Afghanistan. So the point is that the current payment system may be, not optimal, and so the Afghan government is thinking about shifting to paying teachers’ salaries through mobile money like M-Pesa, if you’re familiar with that or it’s like Venmo, but in developing countries, they’ve had it for many years now.

Robert Wiblin: It’s probably better than Venmo, right?

Ofir Reich: Yeah. It works on a feature phone as well, which Venmo doesn’t so, added advantage. So they want to pay teachers this way and see the benefits of this whether it works better in reducing these transaction costs and a lot of these things and so the research team is evaluating that effort.

Robert Wiblin: Skipping back to the tax evasion case. What kind of data do you have to work with there when you’re trying to identify these fraudulent companies?

Ofir Reich: First of all, we’re working very, very closely with the government on this so this is very much a partnership, which is a good way of doing these things if you want something to actually happen in the world. And we have the universe of value-added tax returns from the state in India. So we have every return that every company filed for the past few years. They file every quarter and so we know who sold to whom for how much, so we have the full network of these different companies and who interacted with whom, at which time period and so on.

That’s one piece of it. If you’re familiar with machine learning, in order to make a prediction, you need what’s called labels. You want to predict fraudulent firms and so we need to identify the difference between fraudulent firms and firms that are not fraudulent. So we have a set of firms that the government has identified in the past as being fraudulent because they have this sort of current, manual review process where they inspect these companies. Then, they go to inspect the company and they find that there’s nothing there. This company only exists on paper.

So we have those as our labeled set and that’s what we use for our predictions. Hopefully, we would continue to work with them to have them act on our future predictions. So we get the data and then produce further predictions to have them inspect these future companies and then have those feed back into our system to improve our system.

Robert Wiblin: How different is a complicated machine learning approach to just seeing what characteristics are correlated with a company being fraudulent in the sample that you have? Is it much more successful and is it that much harder to grasp what’s actually going on?

Ofir Reich: So it’s not always simple to know. This data set, though it might seem very comprehensive, doesn’t tell you very intimate things about this company, right? It’s sort of very-

Robert Wiblin: Sparse.

Ofir Reich: Yeah, I mean, it’s all anonymous, right? You know who sold to whom, but you don’t know what these companies do or what their business is. We don’t even fully understand all the mechanisms by which these fraudulent companies act and how exactly all this works. Do they sell to each other? Do they have a complex network? We don’t know all these things. I definitely think domain expertise is crucial in this.

And I think we’ll learn more as we go on, but I don’t think it’s as simple as saying, oh, okay, there’s this single indicator and you just look at that and that’s very correlated. Again, get a little technical, we can see that when we add these more complicated machine learning approaches and added, what’s called more features or more different features of our data and all these sort of different characteristics of these firms and their behavior, we get much better results.

Robert Wiblin: So is it already useful, the algorithm that you have?

Ofir Reich: So I’m always kind of skeptical and cautious. I think for the standards of many machine learning applications, it is useful in the sense that we can see that our predictions are good on our data, but I want to be cautious about that and say that I’ll only really believe it when we actually get data and then produce this list of recommendations of companies to inspect.

And then we inspect them and see that a percentage of them are bogus. It’s like the trick of the magician giving you the envelope and then you open it and it’s your dollar bill that you signed in there, right? So we have to give those and actually see that it actually holds true.

Robert Wiblin: What are some other interesting things that CEGA is doing? Are there any projects that stand out?

Ofir Reich: So it’s important to say that a lot of these projects are carried out by CEGA affiliates that are professors at various universities and so we have over 70 affiliates, again, for this global poverty research [crostalk 00:14:55]-

Robert Wiblin: But most of them aren’t in your office?

Ofir Reich: Yeah, I mean, almost none of them are in our office. They’re on various universities, not even in the same university. We’re based at UC Berkeley, but they’re along the west coast. They do a host of interesting research projects from better measurement of poverty, from using remote sensing from outer space, to a lot of huge, large scale RCTs, randomized controlled trials, such as evaluating the effect of voucher systems in India is a project that a CEGA affiliate Karthik Muralidharan has performed, I think, is excellent. And so there are a lot of these very exciting research projects by different people.

Robert Wiblin: How much impact has your work or CEGA’s work, as a whole, had?

Ofir Reich: So that’s a hard question. I could answer more for my work since I’m very closely monitoring it, of course, because I always want to sort of direct myself to the most impactful place. About CEGA, maybe I’ll start with that, I think attribution is difficult, right? I mean, it’s hard to know … They’re very influential people, there are ideas that have changed, but it’s very hard to know that research changed this specific policy, this specific idea.

It’s very hard to know that … Ironically, we are the people that do RCTs because we say that attribution is difficult, right, but it’s hard to do an RCT of like, half the people will produce academic research and half won’t and then we’ll see how that works out. So I think we’re forced to use some anecdotes. We have some anecdotes of impact, but attribution is very difficult and in the end, it comes down to some people thinking, yes, top economic research advancing knowledge in these fields has given us a lot of understanding and insight into what’s going on and some people being more skeptical, but I think there are a lot of smart people on both sides.

Robert Wiblin: Can you think of any cases where it seemed like funding was redirected based on your insights or policy was changed in some way?

Ofir Reich: There are a lot of examples from the broader RCT movement. For example, from J-PAL, I can give an example from India. There was research by, again, the same affiliate … different research by the affiliate I mentioned before, Karthik Muralidharan. He was at UC San Diego and he did research about a system that they had that uses … in Andhra Pradesh, in India. They use biometric smart cards to ID people, to identify them and also have them authenticate when they receive payments from the government.

So there are two different welfare payment programs and people had to use these biometric smart cards so they didn’t just produce a paper ID or something. They had a smart card that contained their biometric information so a fingerprint and they had to put in the card and then put their finger on a scanner and only if it was them … you know the funding was-

Robert Wiblin: Dispensed.

Ofir Reich: Yeah, it was dispensed. And so they did this reform and this was evaluated in a huge RCT over several districts in this Indian state of tens of millions of people, and they found that there was what’s called leakage, which is a euphemism for corrupt officials stealing some of the money intended for beneficiaries, has been significantly reduced by millions of dollars, which is much more than the program costs.

And so Karthik tells this story that they were in a government meeting with very senior officials, and people were saying this program sounds good but in fact, I know a story from my village where this person wanted to get the benefits, but they didn’t. Or this other story where people say this new system takes them longer and so they’re able to produce this evidence of saying, look, we surveyed thousands of people and it takes less time, it reduced leakage, more beneficiaries get what they wanted and what they deserved, and also there’s overwhelming support for this program, right?

There’s over 90% support for this. This was able to beat some of these anecdotes or maybe vested interests if you’re more cynical, to say, look, this is something we should pursue. So examples like this are examples where we can say with a little more confidence, there was impact, but again, it’s impossible to know the counterfactual.

Robert Wiblin: In any specific case.

Ofir Reich: Yeah. And nobody’s more strict about that than development economists.

Robert Wiblin: So how did you end up working at CEGA?

Ofir Reich: So somebody who studies employment and social networks told me that in any context that you look, about 50% of jobs are through social networks. It’s not through all these assignment mechanisms so I found it through social networks. When I decided I wanted to work in global poverty, I did a lot of research on my own to see what are people saying, what’s the most effective thing. And I found that a lot of the people that were saying very smart things about world poverty were these people at J-PAL and CEGA, a lot of these development economists.

Robert Wiblin: What kind of people are you thinking of? Chris Blattman?

Ofir Reich: Also, Esther Duflo and Abhijit Banerjee. Poor Economics was a book that was very influential for me. I highly recommend it. It’s very well written. It’s a great exposition, introduction to development research and I felt like these people are saying … they seem like the most thoughtful about it, right? They don’t seem to have an ideology driving what they’re saying. Where they’re like, free markets are always best or it’s only more aid that’s going to help us, we just need more money. They said, well, let’s take an approach that measures these things and then we’ll see.

And I felt like they had a lot of very good insights and also seem integrated. So I applied to a few of these organizations, but what eventually came through is I spoke to a person who’s a professor of economics, who’s Israeli that I was introduced to through someone. And he introduced me to another professor who’s a J-PAL affiliate and she introduced me to a professor called Joshua Blumenstock who was at the University of Washington, UW, at the time and now is in Berkeley. And I spoke to him and because I had more of this sort of data science background, which is more similar to the work that he does, he said, “Well, there’s this position opening up at CEGA if you want to apply for that.” And I applied for it and I got it.

Robert Wiblin: What about similar places you could go to if you wanted to leave CEGA? What’s the range of places doing data science and development work?

Ofir Reich: So I’m always afraid that I’m not looking at all of the other places because it’s always much easier to be exposed to your vicinity of like-minded people and organizations and so on. My impression has been that there isn’t a whole lot at this time. That’s due, first, to lack of data because not everything happens on digital devices in the developing world.

Many things happen in person and so it’s not like you have data about every transaction, as you can imagine, the cases that I mentioned, where you might have digital, electronic data about a lot of transactions that are important is pretty rare. Almost nobody collects these in developing countries. It’s slowly changing, but it’ll be a while. And so one thing is lack of data.

And the second thing is that if you’ve traveled to a developing country, you know that a lot of day-to-day life is not mediated by information technology, right? People go and work in their fields in rural areas and that’s what they do. They get paid in cash and then they might call their friends with their phones, but a lot of what they do is not that.

So if you want to influence them somehow through technology, you don’t have a lot of endpoints, right? It’s not like in the developed world where many people, a lot of what they do is in front of a computer and they surf the web all the time and you can find them and influence the way that they do things.

So these two things combine to the fact that there aren’t too many organizations where I feel like they need a data scientist now. Certainly not the most effective organizations. That said, I think there’s good work being done, both in academia and in these specific cases where you have a lot of data and you have somebody that’s more reform minded, and I think in these cases, there might be a large impact just because this approach hasn’t fully penetrated there as it has to industry. But if you have the mindset, like many people in the tech world have, you know, data is going to solve everything, then I would say that’s not the case in the developing world.

Robert Wiblin: You might be disappointed in India.

Ofir Reich: Yeah. Just a little bit.

Robert Wiblin: So the other large one, I guess, is J-PAL, at MIT, right? It might be the most similar organization to CEGA?

Ofir Reich: Yes. J-PAL and IPA are sister organizations of CEGA. We work closely together.

Robert Wiblin: What are their main differences if there are any?

Ofir Reich: Sure. So one difference between CEGA and J-PAL … so again, this is talking about the branch of CEGA that’s about development research and not about transparency in the social sciences or the global networks, which are just different activities. For development research, there are a few differences. One is that CEGA affiliates are not strictly economists. J-PAL affiliates, as far as I know, they’re all in economics. So CEGA has a few people, like Josh, that I mentioned that is a computer scientist and some engineers, political scientists and so that is a bit broader.

The second is obviously what I mentioned. The geographical focus that’s on the west coast of the US, so no other universities. So J-PAL also has country offices and IPA, as well. Innovations for Poverty Action. So they also have country offices where they actually implement randomized controlled trials. CEGA doesn’t have country offices and so when our affiliates … some of whom are also J-PAL affiliates, want to implement their field research, if they are doing field research, then they would go through J-PAL or through IPA or through another implementing organization. So CEGA staff deals only with sort of the US based activities of grant making and communications and those things and not the actual implementation of the programs.

Robert Wiblin: What are the benefits of working in a kind of top academic environment like you would have at UC Berkeley? Is that something that people really should be focused on getting access to?

Ofir Reich: I actually think it is. I’ll explain what I think is great about it. So there’s two things that I feel, especially many effective altruists or people that are more inclined to solve these analytical problems and that’s their strong point, could benefit from that. One is that working in a top institution, you really are working with people that are very smart, very capable, very good at what they do, which, if you have worked in various companies with people with various talents, you know it’s very important.

There’s a lot that you achieve with a group of very smart people that you can rely on to do their best and you can communicate your ideas and you gain from their ideas and you know decisions are being made well. So very talented people is important.

The second thing I think is the sort of combination of brand name and existing connections. So if I think of myself two years ago, I think I had similar talents, less knowledge about development but not something that I think would be an obstacle if I wanted to work in a new organization, but I had nowhere near the same connections that I do now from knowing people and many of these people, again, these top … all these projects that I mentioned, I did not approach the government of a state in India.

These were existing connections that people have cultivated over a long period of time and I think people are … they’re better at this than I am and so I’m very happy to have people, who have that strong point, do that and have my work be what I’m good at. So I think these connections are very, very valuable. And using people’s existing connections is a very strong reason.

Robert Wiblin: So you got your current job through networking and, I guess, now with an even bigger network, you’ll be in an even better position to get whatever kind of job you want next.

Ofir Reich: Hopefully.

Robert Wiblin: Fingers crossed.

Ofir Reich: If I want to be the Prime Minister of Ethiopia, I’m not sure how much my network can help but yeah, definitely more opportunities now.

Robert Wiblin: Are you thinking that academia, that might be different from working in a nonprofit, just delivering an intervention that perhaps the network or the intellectual standards might not be as high as if you’re at a great university?

Ofir Reich: I am concerned about that. Again, I don’t know the nonprofit sector very well. My impression is that some organizations are more dedicated to impact evaluation, to monitoring, to these things and some are less. As a result, some are provably very effective and some are not. I think I would definitely be concerned about working in an organization that goes on gut or goes on things that are sort of not quantitative, results-based inference. All these things that you want to be sure that-

Robert Wiblin: Empirical.

Ofir Reich: Yeah, empirical. To be sure that you’re having the impact that you would want. I think some are. So people have diverging views about who’s as recipient of a project or beneficiary or a government in a developing country, who’s better to work with. I think there are advantages each way. I think some people say they trust us because we’re academics. We’re not, for example, I don’t know, USAID, that might have some other ulterior motive or something like that.

They know we’re in it for the knowledge or for the benefit. Some people say that academics are a little too … They’re not stable enough, right? They go, you complete the project and then the academics don’t stay for many years after that and so I think there’s ups and downs to both.

Robert Wiblin: I guess if you went to a nonprofit, you might find people who excel in a different domain like operations or implementation and things like that, but it would take you away from the academic track potentially.

Ofir Reich: That’s also true. I mean, if you’re pursuing an academic track, then you probably want to be around academia. That’s how the system works. You need recommendation letters. You need the network and that sort of stuff.

Robert Wiblin: So you and CEGA are also interested in reforming problems in academia to try to make, kind of published papers and their results more reliable sources of information that people can really trust. Tell us a bit more about that.

Ofir Reich: Sure. So that’s not my main focus area, but I’ll tell you what I know. I think this is extremely important. So if you’re familiar with the way scientific research works internally, then you know that there are a lot of these practices, in social sciences but also in other sciences, that are a little problematic. For example, results tend to be published if they are statistically significant, meaning that they couldn’t have happened by chance. But the statistical significance can sometimes be engineered.

One way that I mentioned is just testing … secretly having 20 hypotheses and then just testing 20 of them so one of them will be with a significance level of .05, which means it’ll happen by chance once every 20 hypotheses. Obviously, it would happen because you tested 20 hypotheses, or on average one. So if you only publish that one and don’t let anybody know about the other 19, then you create a result that’s published, but obviously, it’s just false.

So this, about, I think, 10 years ago, there started being a discussion in scientific circles about this being a very, very present problem. So this is not some edge case, this is a problem that happens a lot in mainstream science and top institutions. A lot of the practices were this way and sometimes people were not even aware that these are harmful practices. They’re just what you did, what you learned. So there started being a movement around this, and a lot of hype.

It was called the reproducibility crisis because people would try to reproduce the same research and, obviously, if they test my hypothesis that I just engineered somehow and they’re going to try and recreate the experiment, they’re not going to get the same result. It won’t replicate. So it’s not reproducible research. I prefer to call it the published results being false crisis.

Robert Wiblin: Makes it very clear what the problem is.

Ofir Reich: Yeah, you’re right. It’s not that it’s about something intangible, reproducibility. It’s about science not producing the truth, which is a big problem. So if you think about it, the problem might be clear, but the solutions are not that easy, right? I mean, how would you know that people didn’t test a lot of different hypotheses? How would you know if people didn’t do whatever, play with their data a little bit in a way that would produce these results.

So there’s emerging research about better practices to do this. One example that I mentioned before is releasing your data from your experiment and also the analysis so that anybody can check that your analysis is correct, at least at the code level and they could also say, “Well, now that we look at the nitty gritty of the analysis, we see that, actually, they dropped 30% of their observations.”

They never reported them because people didn’t answer a specific question and this causes sample selection or this is a problem that would bias a result that they can publish the criticism. You can obviously realize why the individual scientist might not always have the incentive to make these available so we need to create the incentive environment to do this.

The other thing is pre registration. That’s another good practice. That’s where you commit to the analysis that you’re going to do beforehand, before you collect the data, before you do anything.

Robert Wiblin: So you can’t then gerrymander the analysis to get just whatever result you wanted.

Ofir Reich: Exactly. Or if you do, people will know because you didn’t even list it or you listed it number 50 in your list of analyses that you were going to do. So people will say, “Wait. What about the first 49? What happened?”

Robert Wiblin: Don’t look at those.

Ofir Reich: Don’t worry about those. Oops. So that’s another practice that would lead to results being more trustworthy. There are a lot of other standards around how you should keep your data and how you should keep your code and how should you do significance. Maybe journals should do results blind reviews, which is, for example, suppose there really isn’t an effect, right? Suppose I test if there was a very strong example of whether canvassing changes people’s minds. So there was research that ended up being completely fraudulent. Somebody made up the data, if I remember correctly.

Robert Wiblin: I think that the original research was completely made up or that the surveys weren’t done, but then someone reproduced it anyway or they found it, in fact, was a true result. If they’d done the research, they probably would have gotten the positive result anyway. It is an effect that exists. It’s just that the first paper didn’t do the experiment.

Ofir Reich: Oh, interesting. So I thought it ended up being not … the results not reproducing, but-

Robert Wiblin: Okay. We’ll get back and double check that and I’ll link to it.

Ofir Reich: But anyway, the point was that just because something is a flashy, new, exciting idea, doesn’t necessarily mean it should be published over rejecting a well-established hypothesis. So if somebody were to do an experiment about something that’s very well known in psychology or many people think is important, you wouldn’t want the result to be published or not published based on what they find or the state of the world, right?

So the idea is to say, let’s do a results blind review, where somebody submits their research proposal. They submit everything and then they say what the interpretation would be both ways and people review that paper and decide if to publish it or not this way.

Robert Wiblin: Without knowing whether it has an extraordinarily strong result or a null result.

Ofir Reich: Right. And you then publish it and people know the truth. That’s where it sort of protect ourselves from some of these effects.

Robert Wiblin: So this issue of the science producing false results crisis has come up on the podcast before because it just appears again and again. If you just can’t trust what published papers say then we’re really in a deep hole. But it seems like the trickiest part is giving academics incentives to reform the system because the ones who currently control it are thriving in the current environment, where the incentives aren’t aligned with always producing accurate results.

It’s better just to produce lots of papers that are kind of unreliable but produce interesting figures. Are we making progress on actually reforming the system, given that there must be many people who would probably … it would not be in their interests to see the system changed that much?

Ofir Reich: Yeah, so I think there is progress that we’re definitely making so more and more journals have an open data and open code policy. So they require this of people submitting their papers, so we are improving in that respect. So I think the incentives are slowly changing as people realize how much of a problem this is and adapting and part of the work that BITSS does at CEGA is promoting these with journals, which I think is extremely important.

Robert Wiblin: Do you know what’s driving that change? Is it personal shame or is it a sense that people actually really do want to find true results and so now that they realize that their processes aren’t so good, they want to fix it up or is it coming from the top perhaps that people don’t want to fund research if the methodology is dodgy even if the researchers might prefer to do it that way?

Ofir Reich: Interesting. I’m actually not sure. If I had to guess, I would say that we sort of deceive ourselves about how many of our actions are driven by first principles and deduction, right? A lot of our actions are just driven by sort of what people around us do. So I think many people did not think it’s a … they didn’t really internalize the fact that it’s a bad thing to try 20 different hypotheses until you get one right, that’s how you did science, that’s what they learned. So I think now that the view is changing among scientists, it’s increasingly becoming sort of less admissible and then everything follows from that.

Robert Wiblin: So just gradually, you have a virtuous circle, I guess. So perhaps some researchers, they’re really passionate about this topic of accuracy in science and so they adopt it because they really care about it. Then other people around them notice that they’re doing it and they’re like, “Well, I guess, in principle, this is a better way to do it.” So they just pick it up as well and it kind of spreads like a virus or I suppose a Justin Bieber music video.

Ofir Reich: Exactly like Justin Bieber. I think, also, there’s, with some of these changes and mechanisms, like open data and those things, people can do your analysis again and show that what you’ve done is not right and so there is this … a bit of a fear of your results not replicating.

Robert Wiblin: There’s the carrot and the stick.

Ofir Reich: Yeah, so I think that also might be driving them. It’s a good thing for science.

Robert Wiblin: So speaking of reproducibility problems, some people have criticized randomized controlled trials as being very expensive to conduct each time and also having low generalizability. So maybe you find that one intervention worked at this particular village in India, but then, do you really know that it’s going to work in a different context, in a different country, with a different culture, with different people implementing it and perhaps doing it somewhat differently.

So if you have something that’s quite expensive and perhaps just doesn’t generalize that much to other cases, are RCTs actually that useful and are they good value for money?

Ofir Reich: So I think it’s a very general question, right? I mean, there’s a whole spectrum of RCTs and obviously, the best ones are worth the money and maybe the worst ones aren’t worth the money. I think with respect to generalizability, I can say two things. Again, I’m not the foremost expert on this, but one, is that there has been a meta-analysis by Eva Vivalt, which found that RCTs broadly do replicate.

It’s not a correlation of 1, but it’s a correlation of, I forget, I think more than .5 or something like that. So there is a correlation. It’s not like we do these RCTs and then each one produces a different result and we’d have to do 10 or 20 of them in order to be certain of something. That said, obviously a different context have had different states.

So the second response that I heard from Rachel Glennerster, who’s the director of J-PAL, is that what you should generalize is the principle as being learned and not the effect sizes or the results exactly, right? So if you understand that teaching at the right level to the class is very beneficial because in many developing countries, teachers teach way above the class level, the grade level and so children, broadly, don’t understand much and then don’t improve over the years.

Then, this is a lesson that’s more general and can be applied in different contexts. The specific intervention of separating the class into various things, could differ with the implementing partner or with the context, maybe in some countries it would work. Some countries it wouldn’t and so on, but the general principles might hold and those are the things you should generalize and if you have your theory of change, then you can single out these things that might or might not work in a different context and sort of check the boxes and see that they do work and then implement.

I mean, you could say RCTs are not … of course they’re not perfect, but in some cases they’re the best things we have to really have inference because some attribution problems are very, very difficult. If you wanted to test the quality of private schools versus public schools, it’s very, very hard to know because the kids within those schools are very different. So if you have a policy, you might want to do that.

The last thing I can say is that comparing RCT … I’m not sure what the alternative is, right, because you think, maybe empirical research is completely not of value, right? We should just go on anecdotes and smart people’s opinions, but if you think research is valuable, then a lot of these criticisms levied against RCTs go for all empirical research and you don’t have a lot of … I mean, if you’re doing your data collection, that sort of stuff, which is expensive, it’s going to be expensive in a lot of kinds of empirical research and the generalizability might be the same issues in empirical research.

So I think a different comparison is to compare how much you spend on a program versus how much you spend on research and evaluation. If you look at the sums that are spent on social programs around the world, the sums spent on rigorous evaluation are tiny. So if I’m a country, right, and I’m going to implement this reform in education, because this reform is going to cost billions of dollars, I would want to spend a million dollars or maybe less on a rigorous RCT to show me that it’s really working.

Robert Wiblin: I’ll see if I can get Eva on the show sometime to talk about her paper. And I’ll stick up a link to it because I think it deals pretty well with this question. But speaking of what other approach could you take other than the criticism, I suppose that there is kind of theorizing. I mean, economists have tried to do this. They try to model the economy and some of them think they have reasonable understanding of the principles by which people operate or firms operate.

From that, they can deduce so that they can produce predictions about how different policies will affect the outcome. I guess that has to go hand in hand with empiricism to some extent, but maybe the balance … I saw recently that about 90% of papers in development economics are empirical papers and maybe that’s gone a little bit too far. Maybe it should be more like 20% should be theory-focused. What do you think of that?

Ofir Reich: So, I do know that this is a good point to, again, remind the listeners that I’m not an economist. My formal training in economics is not very wide. So quoting smarter people that are more knowledgeable, I think a lot of economists criticize development economics for not being enough theory driven and I would say that’s what I like about it because I honestly think a lot of these models, they just leave out important things and we don’t know that these things are important.

In some situations they are, in some situations they aren’t, right, and so some problems can be solved without a model at all or they can solve with consultancy approach, which is like, “We’ll be smart, we’ll be analytical about this, we’ll figure it out” and some can’t. So I’m skeptical of a lot of these models and I think if you look at these predictions for a lot of these models, these are not predictions where somebody would say, “Well, there are these two opposing forces. One says people are risk averse. On the other hand, people want to invest to improve their utility so what will they actually do. Well, we don’t know. It depends on how strong these are in the specific context.”

I think what drew me to, again, sort of like the development economists that are in the mainstream today is that a lot of them didn’t seem very ideological about the theory and we were very empirical. We go, we talk to people in the villages. We see the problems. We research and we understand the context very well, but we also are rigorous about the methods and we don’t let general theory guide us because it often misguides us.

Robert Wiblin: There was recently a debate between Lant Pritchett, who is an economists at the Center for Global Development, and Chris Blattman, who’s a development statistician. I’m not sure what his affiliation is. Lant Pritchett was arguing that he thinks kind of the biggest gains in development are really to be had at the macro level with big policy reforms. He points to the case of Deng Xiaoping reforming Communist China to be somewhat more market oriented and allowing foreign investment and allowing big manufacturing industries to grow and people to earn profits and so on.

He thinks that the gains there were so large that they really kind of swamped the gains that you might get from testing individual interventions and then scaling them up, given the kinds of money that’s available. So to him, he thinks maybe doing more RCTs and just trying to move a bit more or less money towards different social interventions, might be a bit of a distraction. But Chris Blattman, who does this kind of research push back on that. Do you have a view on this debate?

Ofir Reich: I listened to it, or read it. I thought Chris Blattman’s response was very thoughtful and he said, yes, growth is very, very, very important and I agree. If I could push a lever and accelerate a given developing country’s growth by 1%, I would give a lot of money to get in the room that has that lever, but we’re not really sure unfortunately on how to do that.

What Chris says is that it’s hard to link the day job of a growth economist to these changes. Maybe we could have some reforms that are useful. I do think, from an effective altruism perspective, finding reforms that you think are very useful or there’s good evidence to say are useful and then strongly advocating for them in poor countries could be an extremely effective thing to do.

I definitely agree with that part. The part about figuring out growth. It could be that growth is made up of all these different good policies that we do separately. Growth is not a single atom, but it’s composed of many of these good policies. So figure out good policies. You think institutions are the problem, there are a lot of good development economists that research institutions in developing countries and try to figure out what are the efficiency costs of corruption, of red tape, of these different sorts of things, so I definitely think that’s important as well.

Robert Wiblin: I’ll stick up links to both Lant’s and Chris’ various interviews and blog posts about this so people can have a listen. It’s very provocative regardless of where you end up coming down. So changing tack, what is it that drew you to apply effective altruist ideas to try to do as much good as possible with your career?

Ofir Reich: Often people ask me this question, I say, “Moral absolutes,” and I assume most people would laugh so I can explain what appealed to me and so I think … I think two main moral principles that I think are true of our morality of our ethical system is that one, there’s a sort of symmetry between people. You can’t want something for yourself and for other people, want it to be differently. Another way of looking at it is the veil of ignorance or-

Robert Wiblin: Golden rule. Perhaps [crosstalk 00:45:50]-

Ofir Reich: Yeah.

Robert Wiblin: Treat other as you’d want to be treated.

Ofir Reich: Yes. And so if you apply that symmetry to other people in the world, clearly, if a poor African can decide what I should do with my finances, they wouldn’t decide, yes, I should keep all of them. I should just do whatever it is that I do and live in my happy family and everything. They’d want me to do something about it so that’s one.

Then the other thing was that I felt like there was no moral distinction between action and inaction. So you’re responsible for your actions and deciding not to think about a problem or not to engage with that problem is a decision. It’s an action you take and you’re responsible for the consequences of that action.

So if I decide to spend all my money on cars, I’m responsible for the action of not spending that money on something that benefits other people. And if I spend my career on working for whatever trading company that I think doesn’t really improve the world and only keeping the money for me, then I’m responsible for that decision.

Robert Wiblin: So you don’t even think that the action/inaction distinction really makes sense in principle. It’s just a spurious idea.

Ofir Reich: Yes. Exactly. When I thought about it long and hard, I figured out that there are many cases where you can’t really even state what’s the action, what’s the inaction, right? I mean, if I had to press a button to keep all my money from going to charity and I press that button, is that action or inaction, right?

Robert Wiblin: That’s a good point.

Ofir Reich: I think also you can also apply the symmetry or the golden rule to that. Would I want other people to use the action/inaction distinction as a justification to do something that would harm me? I probably wouldn’t accept that. Just if they just said, well, but this is what we were doing.

Robert Wiblin: It sounds like you’re pretty sympathetic to utilitarianism. Would that be fair to say?

Ofir Reich: Somewhat. Yeah. I think there’s-

Robert Wiblin: Some issues.

Ofir Reich: … some issues obviously, but I think I like the sort of smaller set of assumptions that I indicated because I think they would be agreed upon by many people, whereas the sort of utility thing where you start doing math and summing things. A lot of people would be averse to that, but they might agree to these sort of principles. But I do think there’s a lot of merits to that. Yes.

Robert Wiblin: Interesting. So if you don’t buy the action/inaction distinction and you think that you should treat people impartially, kind of so you should do as much good as possible, why choose to work on poverty specifically rather than, perhaps, some other problem in the world?

Ofir Reich: So first of all, I would say that I think if somebody else is working on different problem that they think can do more good to many people, that’s great and I’d never be critical of that. I do think part of the reasons that I work in poverty is that, you should also consider what you are inclined to do and want to do. So I like being in developing countries. I traveled extensively in developing countries and I loved it.

I enjoy working on that. It’s rewarding to me in a sense and so I think I chose global poverty, in a sense, before I chose effective altruism, which you know is not the most perfect principle, but you also have to be honest with yourself and know yourself and so that’s what I did. One distinction I feel very strongly about is that our moral circle doesn’t just encompass people that are close to us or are in our own country.

If you think I have the same amount of obligation to somebody in my country as I do to somebody in another country, then pretty quickly, you realize that you should work for people in other countries just because poverty is so much more dramatic and so much more suffering is involved. People say, well, you know there are poor people … I come from Israel, so people say there are poor people in Israel as well and that’s certainly true and working for those people is very important, but the children of those people don’t die from malaria.

So I do think the problems are greater and there is more you can do. There is an amazing opportunity to improve the world in poor countries. So I strongly feel if you think every life has equal value, you should be working where there’s the most suffering and the most opportunity.

Robert Wiblin: So data science and machine learning are two of the paths that we really recommend to people. In your experience, is it a good career? Are you glad that that’s where you specialized?

Ofir Reich: I love it. I think it’s a great career. Everybody should be data scientists. No, I think it’s great for me. I think if you’re inclined to that kind of work, you might enjoy testing things out for yourself with a little code. You don’t have to enjoy building systems, but you enjoy getting results quickly. You enjoy looking into problems, getting to know them deeply like sort of analytical mathematical statistical problems, modeling them. That’s something I very much enjoy.

I think if you enjoy those things, then data science has the added appeal of being very much sought after these days. I could be really enjoying my art. Unfortunately, our world doesn’t reward you fiscally for that and so I wouldn’t suggest that to somebody who’s … you can do it either as a hobby. You can do it in many ways. I don’t know if I’d suggest that as a career, simply because that’s the world we live in.

So data science is very, very rewarding in that aspect of being sought after and so you get people take things off your plate and let you do your thing if you’re good at it. I’ve had a lot of fun working in data science.

Robert Wiblin: Let’s say that someone’s listening and they’re an undergraduate and they’re thinking, maybe I want to study data science, but I’m not sure that it’s a great fit for me. Are there any objective indicators that they can use to tell whether it’s the kind of thing that they’ll excel at and enjoy?

Ofir Reich: Yeah, it’s a good question. I think the best test for something is to kind of try it on for size and do something that’s close and see if you enjoy it. For example, if you try to take up some kind of programing in a friendly language like Python, and you just absolutely detest it. You can’t do it. You only want to work with a pen and paper, then maybe it’s not for you.

If you don’t like working in front of a computer every day, all day, then maybe it’s not for you. If you’re really averse to numbers and statistics, obviously it’s not for you and that’s important, right? I mean, you can’t just sort of go around that. You’ll need to work with data a lot. I think if you’re somebody who likes the cleanness of very academic research and the strict modeling and that sort of stuff where reality and all its idiosyncrasies doesn’t come in to destroy your beautiful model, then maybe this is not for you because many times there are these.

You have to deal with real things, but again, there’s also a broad array of data science jobs. For the more theoretically leaning, do research machine learning, find the better model on the train set or two, something very, very practical like produce indicators, being the right hand of somebody who needs to decide policy. So I would say the elements are statistics, and kind of math and research, which is a bit more difficult to know if you like or not, and programming.

Robert Wiblin: What’s the most annoying part of your job?

Ofir Reich: I think I’m having difficulties with a few things and one of them is that your success doesn’t depend only on your skills. It’s not because people want to harm you in any way, it’s just that you’re working with a partner and they’re in government or they’re someone and they have their own reasons and incentives. Sometimes they understand or don’t understand what you’re doing and you could have this brilliant idea.

You solved the analytical problem and that won’t matter because nothing would happen, right? And things being out of your control is sometimes frustrating. I think I’ve also had to adapt to the fact that in the world of academia and I think in a lot of the nonprofit sector, there’s a lot of reporting of what you do. I was talking to somebody who did the opposite transition from academia to data science in a private company.

She said, “It’s amazing. I finish what I do. If I want, I do a write up. If not, I just commit and it’s part of the system now and then I’m done with it. There’s no two years of going around presenting things, changing people’s minds.” There’s definitely a lot of communicating with other people and communicating about your work, which I wasn’t totally prepared for. It’s okay, but I don’t think it’s a highlight of my work.

Robert Wiblin: Yeah. So you’re using your data science skills to solve a particular kind of problem in a particular way. Are there any other particularly useful things that you would recommend that other people who are doing data science work on? Any other problems where the skill set is really uniquely well-positioned to help?

Ofir Reich: Yeah, so I think, again, from an effective altruism perspective, you have to remember, data scientists of the world, that our skills, as opposed to the skills of some artist or others, are very much rewarded in the private market in the developed world. So you could do a lot of good by going, getting a high earning job and giving a lot of money to people that are very, very effective at doing these things that are great for the world.

And it’s not necessarily that your skills or my skills are the best positioned in the direct work and I think you should be wary of thinking, “Well, if I get a high paying job and I have a good life and it’s fun, then I’m probably doing something wrong, right?” I mean, I think you don’t have to sacrifice a lot to be the most virtuous, I would say, or to do the most good.

You have to remember, first of all, there is this option and if you enjoy what you’re doing and you’re earning a lot of money, seriously consider giving a lot of that money if it’s still rewarding to you and a fun life for you. That said, I think, on global poverty, you should look for places where there is scalability, right, so larger innovations. You’re not going to … doing data science on one village that has one specific program, you can optimize that program by 10%, but it’s not going to pay your salary.

Or it might pay your salary in a system of perverse incentives of donors and everything but it’s not a good thing for the world. So you should just look for large systems where data exists and where the relationship exists and people are willing to change. I think a lot of the promise is working with governments because they have these large data sets sometimes.

At least, they have the ability to create these data sets. They have the ability to change things that are very important and then we would discuss growth and institutions so they have the ability to change those things. I think if you have the opportunity to get, so to speak, very close to the fire where these very important and large decisions are being made and contribute with your expertise then that’s a good thing.

I think the last thing I would say is that you should always ask yourself am I doing something that a local person could not have done. Is it really needed to have this very high end expertise here to improve things because wages in developing countries are very low. So if you can get a hundred people to do something manually, maybe you’re better off not having a data science project.

Robert Wiblin: Speaking of other things that you can do, what kind of stuff did you do in the Israeli Army? Is that something you can talk about or is it all classified?

Ofir Reich: I can tell you but then I’ll have to kill you.

Robert Wiblin: That sounds like it’s worth it.

Ofir Reich: I can’t discuss the specifics of the work but I can say that I did mathematical research and a lot of data research and algorithms to solve these research problems that would last from weeks to months and would be very difficult and sometimes impossible. We had a slogan that says, “What’s very hard? We’re already doing it. What’s impossible? That’s going to take a long time.” Right?

It was very much this sort of atmosphere of solving very, very difficult research problems with a team of other researchers. So you might not think about it when you think about an army, but the Israeli Army, because of mandatory service, can choose from basically the entire cohort of kids that turn 18 that year and become eligible to be drafted. So what you would rightly imagine is that an army doesn’t have a huge demand for mathematicians, and so you can get a lot of the very best mathematicians in the country — if you find them — to come join this unit and so there are a lot of extraordinary people, extremely smart people, extremely capable doing these things.

I both got training that’s worth tens of thousands of dollars in salary because before … I studied physics and mathematics before going to the army and I spent six years there and that training was invaluable because if you just study something like physics and mathematics, it’s not immediately practical. You can go into academia or you’re going to have to do additional training in order to be a data scientist, but working there on learning statistics and learning how to work with data really taught me a lot of the skills that I use today.

But also, these more, I’d say intangible skills of how to do research, how to approach a difficult problem and so to speak, bang your head against the wall until the wall breaks and how to pursue … say okay there’s another tool that I would need. I’ll develop this tool or I’ll learn this skill and also just having sort of a sense for data. Just knowing, oh, well this seems like something’s off here. This seems like it’s going this way. Let me generate hypotheses and test them with data. So a lot of that, I feel like was great training, both professionally and I also had a lot of fun. I had it for six years. It was very good.

Robert Wiblin: So let’s shift gears now and try to dig down as specifically as we can to get advice that people listening could actually use to go into data science or otherwise work in global poverty in an effective way. Where should young people start if they want to work in solving global poverty in a similar way to what you’re doing?

Ofir Reich: I think I might be unusual, in that I didn’t start out wanting to work in global poverty. I studied and then I had my job in the army, which was not focused on global poverty at all. Then, I traveled and then I decided I want to get into it. Then, I still worked for a private company that wasn’t … I mean, I think it was doing good for the world, but not the kind of good that I want to do.

So I think there’s a lot to be said for starting in the private sector for data science. I think the work that I’m doing today might be more impactful, but it’s not building up my skills as well as working in a private company and there are a few reasons for that. One is that there are not many data scientists sort of hanging out in academia and so it’s hard to learn from other people. You’ll often be alone in a project.

There’s a lot of need to sort of … you want to get the results and again, this is reporting the results and maybe publishing and that sort of stuff is not necessarily conducive to developing exactly the data science skills that enable the best problem solving. In a private company, they’re very results driven. They have a lot of data. They have the newest technologies, which is very important.

So I think it’s very important to get a lot of hands on experience if you want to be a good data scientist. I think a good place to do that is in the private sector and I think it’s also more important to sort of get your quote-unquote “credentials” doing that, right? I mean, if you’ve worked for a private sector company and this company has a brand that you can use, then, okay, you’ve had the sort of stamp of approval that really helps going into other things later.

I think I heard Will MacAskill say something similar on the EconTalk podcast and it resonated with me that if you want to work in nonprofit sector, it’s not sure that the first thing you should do out of college is go work in the nonprofit sector.

Robert Wiblin: So potentially, yeah, go and build career capital in the private sector and then cash out in terms of impact later on. Is there a risk of going into the private sector and not building skills that are going to be relevant to the specific research questions that you want to do in order to do good later on?

Ofir Reich: I think, to be slightly controversial, I think people in development are a little high on how much development experience is important, especially for my type of job. I know that if today, I had to hire another data scientist, I’d go for the better person with better data skills instead of the person with more development experience. There is something to be said for development experience.

I think I wouldn’t take a person that I think would never set foot in a developing country, would never understand these considerations, would be annoyed if things are not working perfectly well anywhere outside the private sector, but I think it’s overestimated how much development experience is important for this position. I think that the data skills are the more important part. Unless you think you’ll just get detached completely and be sucked into the private sector, never to return again, which I don’t think is … I was afraid of that, but I don’t think it happens so I’d say maybe-

Robert Wiblin: It’s not much fun.

Ofir Reich: I think it is fun but … and you get used to the snack bar. But I think if you really care about this, I think, maybe maintaining it as an outside interest, reading things, staying sort of up to date with the goings on is enough to sustain you and give you sort of the development experience. That’s not to say that’s not how people are going to hire, so I can’t attest to hiring practices. I can say what I think will make you a better, more excel at this position.

Robert Wiblin: I guess if you were doing implementation in the developing world or something with a lot of culture sensitivities. So trying to figure out and develop an intervention to try to change people’s cultural practices, then it might be a different story. In the case, it might be that direct, hands on experience in the area that you’re working in could be most important.

Ofir Reich: I think it could be most important, but, again, it comes at the expense of other things. If you’re working in a different place, I think it’s important to have an understanding of how real people work in developing countries. I think you can also get that by traveling or being involved with the project. It’s good that you don’t just do it by reading papers. I do think you get a lot, in terms of getting things to happen.

If you’re going to do an implementer part position in the future, that’s extremely important, I think. If you’re going to do a desk job, “crunching numbers”, quote-unquote, like working with data, I think you need the sort of data experience. So if you spent your time making sure that a lot of people do what they’re told, which is very hard and very important, you’re not necessarily getting those skills.

Robert Wiblin: Would you like to highlight any other options or paths in tackling global poverty that are different from the kind of data science that you do that you think are particularly promising, that could have a really big impact?

Ofir Reich: Yeah. So I have two, sort of inklings. One, which is sort of related is that … and it’s a bit cliché, but development economists that are very impact minded, I think could have a huge impact on the world. All of these projects that I mentioned. I did not create these projects. The people who created these projects are development economists or in partnership, of course, with governments. So if you want to take that route, you would have a lot of influence and you can use it to have a lot of impact. That’s one.

The second thing is that I think if you’re a very talented, but also resilient product manager or marketing person, I think you’d be a very cool addition to the world of poverty alleviation because I think modern product development practices have not percolated all the way through and so people aren’t forming the tightest feedback loops, people aren’t doing a lot of A/B testing, people aren’t constantly measuring their outcomes. That’s one thing.

Then if you’re really good at marketing, maybe you can be an incredibly talented lobbyist to lobby for reform and working with governments or very large entities that can implement these very large scale changes. So I think marketing is maybe not what you associate with working in developing countries, but I think those skills could be very useful.

Robert Wiblin: Often people can struggle to get their foot in the door, trying to work on a problem like global poverty, where there’s a lot of people who would like to work to solve global poverty because it’s quite fulfilling and well understood and well appreciated problem. What are kind of the best organizations to work at in your career? You’ve obviously said working in the private sector, but what if you wanted to go and do direct work. Are there any places that it’s relatively easier to get a job when you’re just finishing an undergraduate degree or perhaps a master’s or a PhD?

Ofir Reich: I think getting out of undergraduate or a master’s, if you’re around academia, I think there are both academic projects you could be working on and they’re also organizations that tend to be very leaning towards academia, maybe they were founded by academics or maybe they’re sort of … they do a lot of M&E, so monitoring and evaluation and so on. And so you can get a job in one of these organizations.

I’ve heard someone advise to sort of, similar to the private sector recommendation is that if you go to work in the nonprofit sector, certainly don’t start by setting up your own nonprofit. Also, maybe, don’t start by working in a very small nonprofit that nobody has heard of because later on in your career, people are going to look at your CV and they’re going to see a bunch of names that they don’t know, whereas if you worked for one of these very large actors … again, this is sort of a stamp of approval, to say this person was … they know something.

If you think about it, everybody can just found their own NGO and just write a cool name and it’s very hard to ascertain whether these organizations are doing good work or not and whether this person has acquired skills. So I think, maybe, working for one of large organizations that have a brand might be a good idea.

Robert Wiblin: Going to the opposite end now, which organizations do you think are doing the very best work, where you would be ideal to aim to work at long-term, as your career matures?

Ofir Reich: Yeah. So again, I’m a bit biased in this, but I think, rightfully biased, which is a contradiction of terms. I think you want to work in organizations that are impact minded and results minded and I think that sort of goes through an organization. There’s a large problem with donation based organizations, that they have to produce things that sound good to the donors. They don’t have to be upfront about their failures because they have to not be because then, maybe the funding would be withdrawn and all these things.

So I think some organizations are very committed. I think that the very best organizations that I can imagine if you’re a very smart person wanting to work in this are … I’m very impressed with the GiveWell and Open Philanthropy. I think these are … first of all, they’re very meritocratic in that they don’t require these very specific credentials and all this.

They test you and I think that’s a very welcome approach that’s more aligned with what happens in the private sector. I think it’s a place where you can have a large amount of influence, but, again, this is looking from the outside. Working long-term, I think working in one of these top-notch evaluation organizations, CEGA, J-PAL, IPA, or working on academic projects can get you a lot of these very good connections, good knowledge, and that sort of stuff. There is as glass ceiling in working in academia, right?

You’d be outside looking in, in a sense. So I think you have to be aware of that. You’re probably not going to become a principal investigator on a project if you’re not a professor and so that’s something to be aware of. You would have to go a slightly different route, perhaps. It’s hard to advance to the very, very top of these organizations.

Robert Wiblin: Are there any particular undergraduate courses or post-graduate courses or perhaps, data science boot camps that you would suggest people take a look at?

Ofir Reich: I could mention a few things. None of these are recommendations you should trust in the sense that I learned most of what I learned in the army. If you can go to the Israeli Army, there’s a specific unit. You’d get top-notch, amazing training and good people, but since this option’s not open to most people …. even if it was, I’m not sure most people would be inclined.

I think there’s a good machine learning, introduction to machine learning course on Coursera. That’s very famous. You want to maybe look at Project Euler, which is this cool site that has these algorithm problems that you solve using your programming language of choice and it starts very easy and kind of gets harder and it will sort of give you a sense of what algorithm work is and I think is a lot of fun. It’s a good way to sort of go up. As far as courses, I’d say take these really … the nasty skills courses that you’re not going to do later.

I’m a big believer in introductions. If there’s something that you think, I will never study this on my own, take a class in it, right? If you’re saying I think machine learning is important, but I’ll never bring myself to really learn the fundamentals or that sort of stuff, then go ahead and commit yourself to some kind of class that’s going to be mean and get you to do a lot of things hands on and will get you to learn these things. I’d say go for the hands on things rather than the very theoretical things if you want to do practical work.

Robert Wiblin: Do you think you would be doing more good if you had gone out into the private sector or just somehow tried to make a lot of money and then donated it, instead of doing kind of directly useful work as you’re doing now?

Ofir Reich: So my best answer to this is that I’m constantly unsure. And that currently, if you look at the results that I’m sure I have contributed to and the time that I’m working, I would not make as much impact … I mean, I have not had as much impact as if I was earning to give, but it’s sort of to be expected because earning to give is you get immediate contribution. I think, again, data science is very rewarded in the private sector and so I’m constantly reviewing this. I think as far as expected benefit in the future, it’s still shaping.

I think the tide is starting to turn so that I have more impact with what I’m doing now, but it’s something I’m constantly reevaluating and checking with myself. So I think two things. One, it’s not all sure that it’ll have more impact this way. I’d get a lot of salary by being a data scientist in the private sector. The second thing is that if you are going into this, you should be ready to have a while before the impact starts and it’s not always easy.

Robert Wiblin: I think you probably would be doing a lot more good in your current just than if you were earning to give but that’s a whole other can of worms that we’ll have to deal with in another episode to explain fully why I think that. What’s the biggest downside of the career path you’ve taken so far?

Ofir Reich: I think the biggest downside is living in the United States, living-

Robert Wiblin: Oh, interesting.

Ofir Reich: Yeah, so I mean, I live in the San Francisco Bay Area, which many people think is the best place in the world. I think coming from Israel, where my family is, my friends are, my home is in a deep sense, my culture, that has been the hardest part. Then, some aspects of the job and the differences from the private sector that I’ve highlighted. I think if you’re moving to a different country, then that’s something to consider, but, I mean, it could be different factors for different people.

I think my conclusion from this is to say, burnout is real. You should be aware of what you are not willing to do. If you are an EA and you’re thinking, “Well, I’m going to suffer from this, but it is the moral thing to do. I should do it. I should move to Kenya because that’s where I can work and do the most good.” I admire you. Be aware that it’s hard and it might be harder than you think. And if it means that you’ll be doing it for two years, and then burning out and just ditching the cause completely, then it’s not good.

The Gods of EA would want you to do something that you’re able to sustain, right? Sustainability is very important. So do something that you’d be able to do for a while to have a significant contribution over a long time.

Robert Wiblin: So, in addition to all of the work that you’re doing, you also keep a blog where you go and try to spend time with people who are living completely different lives to you and then, kind of write it up and take photos, is that right?

Ofir Reich: Yeah, that’s right. I have a travel blog. The way that I travel is maybe not the most standard one. When I travel, mostly in developing countries, but not exclusively, I like to go and experience first hand, the lives of people that are normal people, that just lead a life that’s very, very different than mine and I find it both a lot of fun and very inspiring because you learn all these different things that you thought, “Well, I thought humans are this way,” but no.

Europeans and their descendants are this way and many people are not this way and so it makes me reevaluate my own life and the way that the things that I’m doing, but I also enjoy it a lot. So when I get to a different country of place, I find people that seem to be just like normal people living their own life and then I try to join them for a while and it works surprisingly well. They both agree and tell you about their life and I learned a lot and have had wonderful experience with it.

Robert Wiblin: Do you think it’s actually useful for your work?

Ofir Reich: I think it has been. A lot of it I did when I took a year off to travel after the army, but before I started getting into working in the private sector and then getting into world poverty. I think it has helped a lot. I think, for one, it has taught me that this romanticized view of poverty is not accurate. I mean, poor people don’t want to be poor. They suffer a lot. Improvements in their lives really help them. They make them happier.

That has been my anecdotal experience. It has also helped me a lot, I think today, in understanding how things work, which is … It’s a bit hard to convey, but you can get a lot of fanciful ideas about how people make decisions, or what services are useful to people. Why don’t people in rural Africa adopt Bitcoin, right? Wouldn’t that solve all the problems? No, it wouldn’t because you understand their problems better now and you understand how they think about those problems, what they actually do every day.

It’s easy to see people through the prism of what I’m analyzing now. I’m analyzing there, borrowing and saving, right, and so I think about it from this prism. But it’s very useful to think, okay, what is this person concerned about when they get up in the morning? What do they actually do? Where do they go? Just understanding the structure of towns and how that worked and how people talk to their friends so it’s like there’s something irreplaceable about knowing how just everyday life happens in poor countries, which I feel is essential for research, especially since it’s easy to miss. If you’re implementing, you’ll probably encounter it, but if you’re sort of in an office, then you might miss it.

Robert Wiblin: We’ll stick up a link to that blog and people can take a look.

Ofir Reich: Yeah, I would warn our listeners that it’s half Hebrew, half English, but I set it up so that you should be able to follow, even if you don’t speak Hebrew to your great misfortune and pictures are international, obviously.

Robert Wiblin: My guest today has been Ofir Reich. Thanks for coming on the show.

Ofir Reich: Thanks a lot, Rob. It’s been fun.

Robert Wiblin: I hope you enjoyed that episode. If you did, let your friends know about the show!

Thanks for joining, talk to you next week.