Jennifer Doleac on ways to prevent crime other than police and prisons

By and · Published July 31st, 2020

A famous photo of a peaceful protester, Iesha Evans, being arrested by police. Source: Jonathan Bachman.

See all episodes

Enjoyed the episode? Want to listen later? Subscribe here, or anywhere you get podcasts:

…they randomly determined when officers would take the training… so it’s a really nice natural experiment. And they found that this one-day training program pretty dramatically reduced both complaints and use of force.

Jennifer Doleac

The killing of George Floyd has prompted a great deal of debate over whether the US should shrink its police departments. The research literature suggests that the presence of police officers does reduce crime, though they’re not cheap, and as is increasingly recognised, impose substantial harms on the populations they are meant to be protecting, especially communities of colour.

So maybe we ought to shift our focus to unconventional but effective approaches to crime prevention — approaches that would shrink the need for police or prisons and the human toll they bring with them.

Today’s guest, Jennifer Doleac — Associate Professor of Economics at Texas A&M University, and Director of the Justice Tech Lab — is an expert on empirical research into policing, law and incarceration. In this extensive interview, she highlights three alternative ways to effectively prevent crime: better street lighting, cognitive behavioral therapy, and lead abatement.

One of Jennifer’s papers used the switch into and out of daylight saving time as a ‘natural experiment’ to measure the effect of light levels on crime. One day the sun sets at 5pm; the next day it sets at 6pm. When that evening hour is dark instead of light, robberies during it roughly double.

The idea here is that if you try to rob someone in broad daylight, they might see you coming, and witnesses might later be able to identify you. You’re just more likely to get caught.

You might think: “Well, people will just commit crime in the morning instead”. But it looks like criminals aren’t early risers, and that doesn’t happen.

(Incidentally, a different experiment used the discontinuity in daylight savings time to quantify racial bias in police traffic stops.)

While we can’t keep the sun out all day, just installing more streetlights might be one of the easiest ways to cut crime, without having to hassle or punish anyone.

On her unusually rigorous podcast Probable Causation, Jennifer interviewed Aaron Chalfin, who studied what happened when very bright streetlights were randomly added to some public housing complexes but not others. His team found the lights reduced outside night-time crime by a massive 36%, even after taking account of possible displacement to other locations.

The second approach is cognitive behavioral therapy (CBT), in which you’re taught to slow down your decision-making and think through your assumptions before acting.

One randomised controlled trial looked at schools and juvenile detention facilities in Chicago, and compared kids randomly assigned to receive CBT with those who weren’t. They found the CBT course reduced rearrest rates by a third, and lowered the likelihood of a child returning to a juvenile detention facility by 20%.

Jennifer says the program isn’t that expensive, and its benefits are massive. Everyone would probably benefit from being able to talk through their problems and figure out why they make the decisions they do, but it might be especially helpful for people who’ve grown up with the trauma of violence in their lives.

A somewhat similar study of one-day ‘procedural justice’ training sessions for police officers in Chicago found they reduced civilian complaints against police by 10%.

Finally, Jennifer thinks that reducing lead levels might be the best buy of all in crime prevention.

There is really compelling evidence that lead not only increases crime, but also dramatically reduces educational outcomes.

In the US and other countries, there’s been a lengthy and mysterious drop in crime rates since the mid nineties, resulting in crime rates that are now just 25-50% of what they were in 1993.

That drop coincided with gasoline being deleaded. Before that, exhaust from cars would spread lead all over the place. While there’s no conclusive evidence that this huge drop in crime was due to kids growing up in a less polluted environment, there is compelling evidence that lead exposure does increase crime.

While average lead levels are much lower nowadays, some places still have shockingly high levels. Famously, Flint, Michigan still has major problems with lead in its water, but it’s far from the worst.

Jennifer believes that lead affects people’s brains in such a negative way that driving exposure down even further would be extremely cost-effective for its crime-reduction benefits alone, even setting aside broader benefits to people’s health.

In today’s conversation, Rob and Jennifer also cover, among many other things:

  • Misconduct, hiring practices and accountability among US police
  • Procedural justice training
  • Overrated policy ideas
  • Policies to try to reduce racial discrimination
  • The effects of DNA databases
  • Diversity in economics
  • The quality of social science research

Get this episode by subscribing to our podcast on the world’s most pressing problems and how to solve them: type 80,000 Hours into your podcasting app. Or read the transcript below.

Producer: Keiran Harris.
Audio mastering: Ben Cordell.
Transcriptions: Zakee Ulhaq.

Highlights

The biggest problems with US policing and the criminal justice system today

I think the problems that are certainly at the forefront of everyone’s minds right now are around the unnecessary escalation of incidents between community members and law enforcement. So maybe someone calls the police for help but, you know, when they get to the scene, they wind up making an arrest that could have been avoided. Or they use force that could have been avoided. And, in general, I think people would prefer less of that if it’s possible. The other problem is related to racial bias in policing. And this isn’t a problem that’s specific to policing or the criminal justice system. We know that racial bias is a problem throughout society in the United States and lots of other countries too. And so this is a problem that is everywhere, and we should not be surprised that it’s also a problem in the criminal justice system.

So I’d say those are the two big problems in the policing space that we’re kind of trying to figure out how to solve. And then there are myriad ways that we might solve them. And so they kind of range from trying to hire better people, trying to train them better, trying to put better incentives in place. You know, there’s lots of discussion about the role of unions and how hard it is to discipline or fire a cop that repeatedly exhibits bad behavior. There’s a lot. There’s a lot we don’t know about how to do a lot of policing better. Even defining like what is a good cop? What exactly is it that we want police to be doing with their time? It’s actually a really difficult question. And certainly one where the evidence doesn’t have much to say to guide us.

Procedural justice training

So there’s a new study that just came out in PNAS that has caught my attention that I find myself talking about a lot: measuring the effectiveness of procedural justice training for police officers. So basically, it’s like a one day training, this program they put together. So not super extensive. But basically really just helping police officers and pushing them to make the process of interactions with citizens feel more fair, and to leave the citizens with the feeling that they were heard and understood and they trust the person who’s making the decision. That they kind of had taken everything into account fairly. That they trust the process, basically. In general, trainings aren’t necessarily easy to measure the effectiveness of because it could be a very selected sample who takes the training, but in this case, they needed to train all 8000 plus officers in Chicago with this training.

They couldn’t do it all at once, so they basically had to do 25 officers at a time, once a month for many years. And they randomly assigned officers to particular times of when they took the training. So it wasn’t just like, you know, the most senior people first, or volunteers first, or something like that. And so what that leaves us with, is it’s a really nice natural experiment where you can compare the behavior of officers who took the training earlier to officers who took the training later and then measure in that way what the impact of the training is on behaviors and the behaviors that they were looking at were things like complaints received from citizens, and use of force by officers. And they found that this one day training program that officers just took once pretty dramatically reduced both complaints and use of force significantly.

And that was on average across all officers. And I have a hunch that if they were to zero in on just the officers that had used more force in the past, maybe they would see a bigger impact for those officers. Because you’re going to have a lot of zeros in there that kind of water it down. And it also seems like the sort of thing that repeated training could be helpful for. So I think there’s no more experimentation to be done there. But I think it’s a really nice example. Both of a program that police departments should be trying, but also that staggered rollout design that they used just because of capacity constraints, right? They couldn’t train everyone at once. That’s going to be a problem a lot of police departments are going to face in any new training program. So using a similar staggered out design where they just randomize who goes first, who goes second, who goes third, I think, is a really nice model for implementing new training programs in a way that then allows us to measure whether they’re effective. So I really liked that study. And I think that their approach is promising.

The impact of body cams on officer behaviour

We have all these randomized controlled trials now, and basically, on net, they found no effects. So in some places they seem somewhat helpful in terms of reducing complaints and reducing officer use of force. In other places, they actually make things worse and force goes up.

The first big study like this in the United States was done in Washington DC, and they just found no effect on anything. And so there are a couple of possible reasons for this, right? Let’s say one is, especially in the current context, every cop assumes they might be on camera all the time anyway, because everyone’s got cell phones and there are cameras everywhere, businesses and whatever else. And so they were already treated, to the extent of having their behavior filmed.

Another possibility is the hypothesis about the officer’s decision-making in that moment is wrong. It’s not necessarily that they’re like, “Well, I know I don’t need to use force here, but like, I want to anyway, and nobody’s gonna notice”. It might be that most force is used because police are legitimately afraid and that could be a training issue, right? Like maybe they shouldn’t be afraid. And there are other ways we could help them figure out how to deal with that situation. But if they’re using force because they’re legitimately afraid for their lives, then putting a camera on them isn’t going to change what they do. They think they’re doing the right thing in the moment. The other possibility is that we now have all this footage, but because of the fact that we’ve been having this conversation about how we can’t fire anybody, there really isn’t much accountability.

Police effect on crime

If you increase the number of police officers by 1%, we can generally expect somewhere between a 0.1 and a 1% reduction in crime. So that’s a pretty big range, but it’s negative. And it’s pretty big. I think the way to think about this is just in general, hiring more police officers is one of the most cost-effective ways to reduce crime. And if we were just looking at the cost of officers and the social cost of crime, then our cities are massively underpoliced, not overpoliced. Like it would be cost-effective to add more officers. And it’s really only once you add kind of the additional social costs that we haven’t done a very good job quantifying them yet, that you get to a point where it’s like, “Oh, maybe that’s not right and we should be cutting police forces”, but that’s fuzzier in all our minds.

I think that the general consensus is that (the large impact of more police is) mostly a deterrent effect. So it’s not necessarily like catching people after the fact and locking them up, although there is a crime reduction effect from incapacitation as well. Like if you put someone in prison, they can’t commit more crime out in the street. But most of the effect, I think, in most studies that are able to figure out what the channel is, it’s coming from deterrence. So, you know, there are all of these randomized controlled trials of like hotspot policing, where we put a cop on this corner, but not on this other corner over here. And we randomized where we put them. And those studies routinely find a big drop in crime on the street corner where you put the cop.

I think a lot of crime is just like crime of opportunity. You know, it’s not carefully premeditated. It’s like, “Oh, I see a laptop left on that park bench. That person doesn’t seem to be around. Maybe I’ll grab it”. And if there’s a police officer standing right there, you don’t do it.

The effects of DNA databases

If your profile’s in the database, it can be frequently compared with crime scene evidence. And so the reason we might expect a deterrent effect is that you know you’ll be more likely to get caught if you offend again. So basically if the database matches you with crime scene evidence, then that’s a lead that police might not have had otherwise. So it’ll identify you as a suspect in a crime if you might not otherwise have been a suspect. So we see this big deterrent effect, but the DNA database itself could actually lead to an increase in likelihood that you get caught. I think that’s the whole point. And so if you just sort of look at the data we might’ve expected, if it had no effect on recidivism, we would expect an increase in the likelihood of being reconvicted, right, because you’re just more likely to get caught.

And so the fact that, especially in the US context, where I had to just look at the net of those two things, I still see a significant negative effect means that that’s an underestimate of the benefit. So I find that in the United States, being required to submit a DNA sample to the database reduced the likelihood of a new conviction within five years by about 17% for serious violent offenders. And that was statistically significant. It reduced the likelihood of another conviction by 6% for serious property offenders.

Articles, books, and other media discussed in the show

Jennifer’s work

Peer-Reviewed Publications

Working Papers

Published papers by other people

Everything else

Transcript

Rob’s intro [00:00:00]

Robert Wiblin: Hi listeners, this is the 80,000 Hours Podcast, where each week we have an unusually in-depth conversation about one of the world’s most pressing problems and how you can use your career to solve it. I’m Rob Wiblin, Director of Research at 80,000 Hours.

This is the second of two interviews we’re doing on policing and criminal legal reform in the US — the first with James Forman Jr came out earlier this week.

I chose to interview Professor Jennifer Doleac because I’m a fan of her podcast Probable Causation, where she interviews social science researchers studying crime and crime prevention.

I’ve actually recommended her show here before, as an example of honest and detailed discussion with subject matter experts — something that was hard to find in public until podcasts became popular.

Like all good social science researchers, Jennifer demonstrates a great commitment to scrutinising the statistical methodology papers use, to ensure they’re producing informative results rather than misleading ones.

I hoped that she would have a lot to say on what good evidence is out there regarding how we could best address some of the failures in policing and law enforcement in the United States, and she didn’t disappoint.

I was especially excited by our discussion of ways to prevent crime that don’t involve police or the criminal justice system, and all the pathologies and enormous human costs they bring with them.

First though I just wanted to take a minute to provide some concrete numbers regarding just how many people in the US are in prison, and for what.

If you’d prefer to skip straight to my conversation with Jennifer you can jump forward X minutes in your podcasting app.

We’ve taken most of these out of a report from the Prison Policy Initiative called ‘Mass Incarceration: The Whole Pie’, by Wendy Sawyer and Peter Wagner. It includes some stunning charts that are worth seeing for yourself, so we’ll link to it in the show notes.

As we discussed in the last episode, in recent decades the US has had the highest incarceration rate of any country in the world, at around four times what’s typical globally.

One in every 110 adults is currently locked up, which comes to 2.3 million in total: ~1.3 million people in state prisons; 630,000 in local jails; and 225,000 in Federal prisons & jails.

That rate actually peaked at around 1 in 100 adults back in 2007.

It will come as no surprise that people of colour — who face higher rates of poverty and many other social disadvantages — are also dramatically overrepresented in the nation’s prisons and jails. These racial disparities are particularly stark for Black Americans, who make up 40% of the incarcerated population despite representing only 13% of U.S residents.

While 630,000 people are in jail at any given time, people are checked into a jail 10.6 million times a year, 18-fold as many as are checked into prisons.

Just a heads up that prisons are where people are incarcerated once convicted, while jails are meant to be for those awaiting trial or serving shorter sentences.

Jail churn is particularly high because around three quarters of people in jails haven’t been convicted of any crime and may never be.

Why are so many unconvicted people sitting in jail? A combination of judges decided they’re a danger to the public or might not show up in court, and them not being wealthy enough to afford the bail they’d have to pay to live at home until their trial.

An estimated 13 million misdemeanor charges are made each year, including offences as minor as jaywalking and as serious as stealing something with little financial value. Low level offenses with penalties of under a year account for nearly 25% of the jail population on any given day.

Looking at everyone who actually has been convicted of a crime and is currently incarcerated, 47% are there for a violent crime, 19% for drug crimes, 17% for a property crime like theft and burglary, and 17% for a so-called public order offence like drunk driving or weapons possession.

So while drug offenses account for the incarceration of almost half a million people, and police make over 1.5 million drug possession arrests each year, 4 out of 5 people in prison or jail are locked up for something other than a drug offense.

I was interested to learn that less than 9% of all incarcerated people are held in private prisons, which to me suggests the private prison lobby probably isn’t such a big driving factor behind mass incarceration.

In the last episode James Forman Jr said the most common violent crime people were locked up for was robbery. That was true up to 2016 but it’s now narrowly in second place to murder.

How likely is someone who commits a serious crime to be caught?

Here’s a calculation that can give a rough sense. In 2017, victims reported some 2 million serious violent crimes (i.e. rape or sexual assault, robbery, and aggravated assault). In the same year there were approximately 450,000 arrests for these crimes. So, even if all those people were guilty, the chance of being arrested for a serious violent crime would be at most 22%.

Alternatively we can look at what police report as their ‘clearance rate by arrest’. This deals better with one person who’s arrested on suspicion of multiple crimes, but misses crimes that occur but are never reported to police. In any case, for the US as a whole we get clearance rates of 46% for violent crimes and 18% for property crimes.

Let’s talk for a minute about the length of sentences for different crimes.

There are many stories of people in the US receiving astonishingly long sentences for relatively minor crimes. I heard one recently where someone was given a life sentence for possessing a substantial amount of crack cocaine, because of a three-strikes law in Oklahoma that mandated a life sentence for any third conviction. Fortunately, after spending 15 years locked up, their sentence was recently commuted. Tragically, their daughter was young when they were locked up and so they have completely missed seeing their child grow up.

But while such cases shock the conscience and are far more common than they should be, it’s important to keep in mind that they’re not typical.

Looking at a report on people leaving state prisons in 2016, we see that the average time served for drug possession was 15 months, and the median 10 months.

For a more serious crime like robbery, which is the taking of property by force or using the threat of force, the average time served was much higher — 4.7 years with a median time served of 3.2 years.

Scanning some others, the average sentence for non-violent theft was 17 months, drug trafficking 26 months, burglary also 26 months, assault 30 months, rape or sexual assault 6.2 years, and murder 15 years.

Adjusting for some sampling bias in this report might lead to estimates 5-10% higher but hopefully the data still gives you the right idea.

I took a quick look at sentence length and they seem to have increased up to 2000 but been fairly flat since then.

I’ll leave you to judge whether those sentences seem to high or low. But sadly those numbers suggest to me that we can’t end mass incarceration just by commuting the most extreme and perverse sentences.

Alright, I hope you find those facts and figures provide some helpful background for this episode and the previous one with James Forman Jr. As always we’ll put up links to all the sources we’ve drawn on.

Without further ado then, here’s my conversation with economics Professor Jennifer Doleac.

The interview begins [00:07:25]

Robert Wiblin: Today, I’m speaking with Professor Jennifer Doleac. Jennifer completed her PhD in economics at Stanford, and is now an associate professor at Texas A&M Department of Economics, as well as Director of the Justice Tech Lab, a senior fellow at the Council on Criminal Justice and a research affiliate at the University of Chicago Crime Lab. Her research focuses primarily on crime and racial discrimination and her papers have investigated discrimination against African Americans in online secondhand markets. The impact of Ban the Box policies, which are designed to help previously incarcerated people get jobs and the impact of DNA databases on recidivism. She also hosts the podcast, Probable Causation, which focuses on empirical research into crime and criminal justice. And it goes into social science research in a level of detail you can rarely find anywhere else, at least spoken, which has made me a big fan and in fact, I’ve recommended that people listen to it on this show before. So, all that said, thanks for coming on the podcast, Jennifer.

Jennifer Doleac: Thank you for having me.

Robert Wiblin: Obviously, we’re talking about racial disparities and crimes committed by police and crime in general at a pretty sombre moment, when they’re very much in the news and, and very much on people’s minds. I’ve had a bit of an amateur interest in this topic for a couple of years, I guess, because frankly, it’s pretty hard to look at the kind of injustices and cruelty that one sees me ted out in the US criminal justice system, and not be outraged and drawn into following it somewhat.

Robert Wiblin: That said, I’m not American and I don’t live in America. And this certainly isn’t my area of expertise, which means that there’s a pretty good chance that I’m misunderstanding the situation in the US in some way. And I probably have some pretty incorrect views on this topic somewhere. So I’m hoping you might be able to guide me through this over the next couple of hours, and also perhaps fill in the half of the audience that isn’t from the United States and maybe is pretty out of the loop on these issues. Does that sound good?

Jennifer Doleac: Sounds great.

Robert Wiblin: Fantastic. Alright. Well, we’ll get to talking about what evidence there is on the best ways to improve the US criminal justice system and how listeners might be able to contribute to that. But first, as always, what are you working on at the moment and why do you think it’s important work?

Jennifer Doleac: So a bunch of things. I have to keep a list for myself of the projects I’m working on so I don’t forget about things. So one project is with Anna Harvey, who’s a researcher at New York university and Amanda Agan at Rutgers looking at the impacts of prosecutor reform efforts. So we’re currently focused on policing in the US in this current moment, but prosecutors had been a big focus of attention before this, and we are getting really amazing data from some prosecutor offices to understand what the role of prosecutors is, and just sort of applying their discretion in terms of who gets convicted or charged in the case of prosecutors with low level offenses and what the long run effect is on those individuals. And then the offices we’re working with are actually ones that have been making pretty big reforms in what prosecutors are doing. And so we’re trying to figure out whether those reforms have the benefits they’d hoped for.

Robert Wiblin: Right. I guess apart from that, there’s an explosion of interest in criminal justice reform just now. And I imagine you’re fielding a lot of phone calls from people. Yeah. What are people asking? Is it just the media or are politicians asking what can we do? Are police departments calling up asking for advice?

Jennifer Doleac: Yeah, it is definitely an interesting moment. I mean, I think as someone who’s worked in this space for a little while now, the topic of police and police brutality, and use of force, and racial bias in policing has come up repeatedly over many years. But the momentum for change right now has really been much, much bigger. And it seems to be lasting longer than anything that I’ve seen in the past personally. And so there is lots of interest right now in figuring out what we can do and what both the evidence says, but then just trying to generate ideas and hypotheses that we can test. And so, yeah, talking with lots of journalists, talking with practitioners and policymakers and other researchers, and it’s kind of an all hands on deck moment where everyone’s just trying to… I don’t think anyone’s happy with the status quo. Like even the police aren’t happy with the status quo. So everyone is motivated to move us to a better equilibrium.

Robert Wiblin: All right. Well, yeah, let’s launch into discussing specific problems with US policing and I guess how we got where we are. And after that, we can turn to options for reform and how promising they look and how we might be able to actually put them into practice. And, as I said before we started recording, you’re kind of an empirical social science economics researcher. I suppose I’m keen to know what things we have really strong evidence for, where there’s a smackdown paper that really demonstrates something at least for a specific case, but also just to get general impressions, even if there aren’t papers to demonstrate those impressions. So I guess don’t be shy about editorializing, but I suppose label them as editorializing. All right.

The biggest problems with US policing and the criminal justice system today [00:11:56]

Robert Wiblin: To start with, yeah, can you give an overview of what you think are the biggest problems with US policing and the criminal justice system as a whole as it exists today?

Jennifer Doleac: Yes. That’s a big question. So there are many problems with policing in particular. I think the problems that are certainly at the forefront of everyone’s minds right now are around the unnecessary escalation of incidents between community members and law enforcement. So maybe someone calls the police for help but, you know, when they get to the scene, they wind up making an arrest that could have been avoided. Or they use force that could have been avoided. And, in general, I think people would prefer less of that if it’s possible. The other problem is related to racial bias in policing. And this isn’t a problem that’s specific to policing or the criminal justice system. We know that racial bias is a problem throughout society in the United States and lots of other countries too. And so this is a problem that is everywhere, and we should not be surprised that it’s also a problem in the criminal justice system.

Jennifer Doleac: So I’d say those are the two big problems in the policing space that we’re kind of trying to figure out how to solve. And then there are myriad ways that we might solve them. And so they kind of range from trying to hire better people, trying to train them better, trying to put better incentives in place. You know, there’s lots of discussion about the role of unions and how hard it is to discipline or fire a cop that repeatedly exhibits bad behavior. There’s a lot. There’s a lot we don’t know about how to do a lot of policing better. Even defining like what is a good cop? What exactly is it that we want police to be doing with their time? It’s actually a really difficult question. And certainly one where the evidence doesn’t have much to say to guide us.

Jennifer Doleac: So I mean there’s just a lot in the policing space. Criminal justice, more broadly… So I spend most of my own research time focusing on the issues of prisoner reentry, as well as the role of technology in public safety, and both of those tend to or overlap sometimes. Sometimes not. But especially on the prisoner reentry side, we have incredibly high recidivism rates in the United States. We do incarcerate an absurd number of people in this country. And most of them get out at some point and we’d ideally like to them to reintegrate into their communities in a productive way. And we’re not very good at that. And it’s sort of appalling how little evidence we have to guide that effort too. So I did a study a couple of years ago on Ban the Box, which is a policy to try to help people with criminal records get jobs.

Jennifer Doleac: And that got me really interested whenever I talked to policymakers about something that doesn’t work, and Ban the Box is an example of that which we can talk about more later, I always want to come in with some suggestions about what else could work, so I don’t leave the conversation with “Well, it’s this or nothing, and this didn’t work, so we all just throw up our hands in despair”. And so I started digging into the literature and it was just really amazing to me how little we knew about what works in the reentry space. So I’ve done a bunch of work there and yeah, there’s still still a lot of open questions. But I’d say we don’t yet have a good sense of how to reduce recidivism rates. We don’t have a good sense about how to fix policing. There are a lot of things we don’t know.

Robert Wiblin: Yeah. So I’m curious to try to, I guess, put the numbers on this so that the audience can understand the scale of these different problems. I guess the police in the US kill around a thousand people every year. It’s like a bit over a thousand. And there’s over 2 million people incarcerated in the US which is several manyfold higher than most other rich countries, and one of the highest rates in the world. One thing I wonder is if the killing of people is kind of a tip of the iceberg of problems with US policing? You could imagine that assaults or harassment… it’s a very visible kind of tip of the iceberg because it’s very visible and people end up talking about it a lot. But I wonder, doesn’t the fact that police are able to do that, or that’s not such an unusual practice indicates that there might just be a lot of assaults or harassment that are occurring. And while each of those is much less bad, they could be much more frequent because people are able to get away with it, even more than I guess police are getting away with perhaps unnecessary killings.

Jennifer Doleac: Yeah. I think that’s absolutely right. I think it’s been remarkably difficult to get high quality data on even the number of people that police kill every year. So the data that we do have basically come from efforts by media companies or other nonprofits to collect statistics based on media reports of killings by police. And so you could worry that those are going to be undercounts, especially in certain places where it doesn’t make the news somewhere and then it doesn’t wind up in that tally. So that’s been hard enough. And that’s a situation where there is someone who is dead and the police either killed them or they didn’t. And it seems like a very easy factual thing to figure out. I think the FBI has, I think, announced that they’re going to start collecting that information from police charts going forward in a more standardized way.

Jennifer Doleac: So we will see if that improves. But I think part of the reason that we’ve been focusing on police killings is, as difficult as it is to get that data, and as inadequate in many ways as that data set is, getting standardized, high quality data on any other use of force is basically impossible. You can get data from an individual police department. So I’ve been helping the Seattle police department analyze the effects of their body worn camera program. And so they have been providing data on use of force at various levels. And so working with a particular police department, you can get that information. But there are thousands and thousands of police departments across the United States and it’s entirely decentralized. And so getting a standardized source of information like that is just on the to-do list, let’s say.

Robert Wiblin: Yeah. Do you have a sense of how common harassment of people is by police or lower level assaults? Even kind of qualitative–

Jennifer Doleac: No.

Robert Wiblin: Yeah. I read that when people had looked at racial disparities and behavior by police, that they found evidence of bigger disparities in harassment and assaults than in killings. Have you seen that result as well?

Jennifer Doleac: So in general it’s somewhat challenging to find policy effects, Or to pick up statistically significant differences based on police behavior on the outcome of police killings because while there are far more of them than you would ever want, they actually aren’t that common by statistical standards. And so since it’s a relatively rare event, again, in terms of the statistical power, it’s just really hard to find any sort of significant effect there. And so the outcomes that you’re able to detect effects on are usually much more low level stuff. So my favorite paper on racial bias in policing is by Jeremy West, where he basically uses car crashes on highways in Texas to look at the effects of racial bias by state patrols. So basically, if you get into a car accident, then you call 911 and basically the closest officer is dispatched to the scene.

Jennifer Doleac: And that avoids the problem that, in general, if you just look at information or data on interactions between police and citizens, the police get to decide who they stop. And so there’s a selection problem there that is really difficult to get around in research terms. So in this experiment, he was looking at what happened if the officer that happened to be closest to you that was dispatched to the scene was white or black. And then basically what he finds is if it’s a black driver and a white officer, you’re much more likely than if it were a white driver, to be written up for really low level stuff like an expired registration, or an expired license, or it’s something like little, low level stuff that you know is certainly verifiable. So you can’t say it’s what economists would might call statistical discrimination where they’re trying to guess about you.

Jennifer Doleac: Like it’s either expired or it’s not. But what seems to be happening is that they kind of give the white drivers a pass, but they don’t give the black drivers a pass. And so I think a lot of good evidence that discretion over that kind of stuff is a place where racial bias can easily seep in. What he does find is there’s no evidence of racial bias for felony charges or something like that. And he argues that that’s a place where officers might expect more scrutiny and so they don’t have the power to use their discretion and apply it in a biased way. And so, yeah, I don’t know. I mean, I think there’s some growing evidence and it kind of depends on the context a little bit to really nail down and quantify the causal effect of the race of the person that the police are dealing with on use of force and everything in between. But I’d be willing to speculate here that, in general, racial bias is a problem throughout that spectrum. And it’s really just a matter of quantifying it. And then of course the next step which I, and others have been really pushing for, is for researchers to focus on more is what to do about it. How do we reduce that bias?

Robert Wiblin: Yeah, I guess another category of problem with the criminal justice system that I thought of when I was preparing for this interview is kind of enforcement of victimless crimes. I guess things that I don’t think potentially should be crimes at all. And the damage that that’s done. I guess, in particular, drug laws stand up there. So it’s a drug prohibition. Potentially people having their lives ruined by drug possession charges where, I guess in my view, that shouldn’t be a crime at all. Yeah. Do you have any quantification, I suppose how much damages is done through drug prohibition? Is this something that you’ve thought about?

Jennifer Doleac: So this is definitely something that comes up a lot. So in the US now, a lot of States have legalized marijuana for instance, and possession of especially smaller quantities of marijuana. I don’t think we have any studies yet, not that I know of, of the impacts of decriminalizing marijuana possession on the trajectories of the people who would have potentially been convicted of that crime if the law hadn’t changed. There’s potential that police charged people with marijuana possession when they’re just looking for some low level charge to get you on. And so if that is no longer an option, maybe they charge you with something else. And so it’s actually not obvious that this fixes lots of problems. Often I think there’s a general consensus or conventional wisdom, I would say, out there that a lot of people in prison are in there for these sorts of victimless crimes and low level drug possession. Most people in prison are there for serious crimes. And that decriminalizing drug possession would not end mass incarceration in the United States.

Robert Wiblin: I see. Yeah. How much of a dent do you think it might make?

Jennifer Doleac: Oh, I don’t know off the top of my head. I think some people have kind of, I feel like John Baff does a bunch of work around this and always has some good stats in this space. But the answer is not much of a dent. Like if we want to reduce incarceration rates in any meaningful way, we need to be dramatically reducing incarceration for violent offenders.

Robert Wiblin: That’s interesting. I guess I hear instances, and I guess I’ve heard from people I know who’ve worked in the US legal system, instances of people going to prison for getting long sentences for drug possession charges, or even drug dealing charges that don’t have a violent component. Yeah. How is it then that that’s not a significant fraction of the total incarcerated population?

Jennifer Doleac: Well, so there are certainly anecdotes like that out there and certainly more on the drug dealing side. I think in the drug possession side, when we’re talking about long prison sentences, but most people who get really long prison sentences are violent offenders and there are a lot of them in the United States. And so just when you look at the numbers, most people incarcerated in any given time are not there for drug charges.

Robert Wiblin: I see. Yeah. I guess maybe the people on drug charges are getting paroled earlier. So perhaps they’re spending less…

Jennifer Doleac: Also, especially if we’re talking about drug dealing, I mean, there’s also a lot of violence that can be involved in that, right? So you might also wind up with violent charges.

Robert Wiblin: Yeah. Okay. I guess a different category of failure would be the failure to prevent crime. And I guess if you do international comparisons, simply the US has higher crime rates than other countries. And there’s going to be a whole lot of factors that go into that, but potentially the criminal justice system or policing, just doing a bad job at preventing crime could be one factor. Is there any research on that? Could the US reduce crime rates significantly if it adopted better international best practice?

Jennifer Doleac: So I’m not sure we actually have higher crime rates in general. We have higher violent crime rates than most places. We also have a lot more guns than most places. I’m always a bit reluctant to make international comparisons because countries are different for lots of reasons. And it’s not necessarily just that police aren’t doing a good enough job here. I think we could have a conversation about the ways in which they could be doing a better job, but I don’t think that the differences between the US and Australia or the UK would be all about one policy that’s different or something like that. I think there are obviously major cultural differences that are longstanding and harder to change. But yeah, I mean, I think in general we definitely incarcerate more people in the US and I think there’s a general consensus in the research space at this point that basically if you think about from economics, we have this concept of diminishing marginal returns.

Jennifer Doleac: So the first person you put into prison, the serial murder, you get a lot of benefit from that. You reduce crime a lot if you put them in prison. But the second person that goes into prison, you get a little bit of less benefit and the third person. And so by the time you go past the rate of incarceration that’s typical in Western Europe to the rate that the United States has, most of those marginal offenders probably aren’t gaining as much in terms of public safety. And rolling that back a bit would probably actually have large benefits in turn, at least in terms of on net. We pay a lot to lock them up and we’re not getting any public safety returns out of it. There’s also concern that for people on the margin like that, putting them in prison, it could actually be criminogenic in some way. It could make them more criminal rather than less. Just like disrupts their lives… If they had a job, they lose it. That sort of thing.

Robert Wiblin: Yeah. Would you want to speculate about what fraction of that 2 million it would be sensible to release if the US could get to kind of an optimal incarceration policy? It’s a big question.

Jennifer Doleac: I’m not sure. Yeah. I feel like people ask me this every once in a while. And it’s a really hard question to answer. And I mean, I feel like moving closer to the incarceration rates we see in Western Europe would be a good goal. I feel like there are major cultural differences between the US and Europe. And especially, again, I think the gun issue is one where we might wind up having higher incarceration rates in equilibrium, and it might be the socially optimum level. But we’ll see. I think as an economist, and especially an economist who’s very focused on policy, I tend to think very much on the margin. And I think pushing us in the direction of reducing incarceration is probably a good move, knowing exactly what the optimal incarceration rate in the US is, is, I think, a bridge we can kind of cross once we’re closer to that margin.

Police misconduct [00:26:53]

Robert Wiblin: Okay. Yeah. We’ll return to a bunch of these problems over the course of the conversation. Let’s talk a bit about police violence/police brutality. Where do we know that people of different races are treated differently by police or courts and kind of how stark are the differences. How do we know or try to measure that?

Jennifer Doleac: Yeah. So to talk about a few studies. So one is that study by Jeremy West that I mentioned earlier, looking at racial bias in the course of traffic fatalities and how police deal with drivers when they arrive on the scene. There’s another paper by Mark Hoekstra and Carly Will Sloan where they look at the dispatch of officers to 911 calls. And it kind of uses a similar sort of strategy saying, “Well, the people who are dispatched to the scene is going to be whichever officer was closest at the time the call came in”. And so it’s essentially random whether the person who shows up is white or black, you know, conditional on neighborhood fixed effects or whatever. And so what they wind up finding is that white officers, especially when you’re focusing on calls from black neighborhoods or majority black neighborhoods, if a white officer, instead of a black officer is dispatched to the scene, then they’re much more likely to use force during the encounter.

Jennifer Doleac: So that’s a nice paper I will say. And kind of on the flip side, Emily Weisburst has a paper using data from Dallas and a very similar strategy. And she doesn’t find any racial bias in arrests. She was looking at arrests as the outcome. So I think that kind of highlights the importance of context and how this is going to vary from place to place. And depending on what kind of outcome measure you’re looking at. Let’s see, we have a bunch of studies from the court context where people try to study this by looking at contexts where defendants are randomized to different judges for the bail hearing, essentially where it’s decided whether they get cash bail, or whether they’re detained pretrial, as well as the judge that would actually hear their case. And so randomly assigning people to be locked up or not would be completely unethical, but randomizing people to different judges happens all the time.

Jennifer Doleac: And it turns out that different judges vary dramatically in the propensity that they lock people up. And so randomizing people to different judges essentially randomizes them to a higher or lower likelihood of incarceration. And so we can use the randomization across judges to see if white and black defendants are treated differently. And they typically are. I mean, again, I feel like it’s pretty standard and unsurprising to find evidence of racial bias in most settings. And so the question is really the extent of the racial bias, to what extent it’s coming from. Is it just like a few judges, a few bad apples versus like the whole system? And then of course the next question that I think is really the research frontier, which is what we do about it.

Robert Wiblin: Yeah. What do you think we’ve learned about criminality among police and police departments over the last couple of weeks? I guess I already had the impression that police in the US were kind of unusually lax about following the law by rich country standards. I guess watching the sheer number of videos of police violence against protestors often seemingly organized as a group that’s come out of the US, it’s made me think that maybe the problem of just police departments as a whole being kind of corrupt and not super interested in following the rules is maybe much worse than I had thought.

Jennifer Doleac: So I have never heard it framed that way before. I’m not sure I would label police behavior as actively criminal.

Robert Wiblin: I don’t know. If they beat and if they assault protesters in a way that wouldn’t be a justified use of force, isn’t that kind of criminal behavior in the same way that it would be if anyone else did?

Jennifer Doleac: So the challenge here is how you define criminal. And unfortunately, police, for better or worse, police have a lot of discretion in when to use force. And so, I mean, in the death of George Floyd, which was the death that sparked all of these protests, the officer was actually arrested and charged with his murder. And I think that it’s very difficult to watch that video and interpret what the police officer did as anything that was remotely justified or necessary in order to defend himself or something. So sure. Yes, there are definitely examples like that, where I think it’s very easy to look at the video and say that that does not seem justified. And then that would not be if the courts agreed. Then that would be outside the bounds of their job description and would be criminal.

Jennifer Doleac: But again, police have broad discretion in using their judgment to figure out what is justified and what isn’t. And so that is something that is very much a topic of conversation right now, which is how do we change that? And I think a bigger issue here is kind of the role of policing unions in this conversation and the extent to which the unions really heavily defend all officers, regardless of whether they seem like they’re doing a good job on the force and should be defended. And so again, that is a big piece of the conversation right now, which is to what extent we can make it possible to fire bad actors in police forces? And that’s very difficult to do right now.

Robert Wiblin: Yeah, because I suppose I’m using a more layperson’s notion of criminality because maybe I’m not thinking about it strictly if you apply… What’s the term, it’s like police can kind of seemingly just do whatever they want and the courts will give them a pass. I suppose I’m thinking maybe from a more moral point of view or commonsense point of view about what the community would think is legitimate use of force. Yeah. I guess one thing that stuck in my mind from a few years ago is season three of the podcast Serial. They kind of went and looked at into policing practice in Cleveland. And they chose Cleveland, I think not because it was the worst police department or something in the country, but rather because they got unusually broad access to the court building to report on criminal justice there. And to be honest, I was just blown away, I guess, as a non-American. It seemed to me like the police in East Cleveland came across as absolute thugs. And it seemed frankly more like an organized criminal organization than a law enforcement agency. Yeah, did you get a chance to listen to that one?

Jennifer Doleac: Yep. Yeah. I mean, I think there’s a reason that these stories are being elevated. I think they’re really important and they need to be addressed. And there’s a reason there’s so much appetite right now for reform in the United States. And I am fully supportive of that and look forward to helping to build police departments that are trusted by… if not everyone, than the vast majority of society. And that’s not the case now. There are also a lot of police officers who are doing a really good job. And in general, hiring more police officers does reduce crime. That does not mean that every officer is good or everything that officers do over the course of the day has a net social benefit. But I think it’s going too far to say that all police departments in the United States are basically just like organized crime organizations.

Jennifer Doleac: Like we’ve got some problems. We’ve got major problems. I don’t want to undersell this. But I think the reason that those kinds of podcasts are important and the stories are told, is because that’s not the standard narrative. And I think it’s really important to point out to people who don’t necessarily have negative interactions with police departments themselves, that lots of people do have negative interactions with police all the time. And so the tremendous value that the podcast Serial did in that conversation was getting someone like you to say, “This is crazy. We need to fix this”. And so that’s a tremendous public service.

Robert Wiblin: But I shouldn’t think of it as typical?

Jennifer Doleac: Yeah. A challenge here is figuring out what is different. I mean the data is so spotty and there’s still so much we don’t know. But yeah, not all cops are bad. I think it is fair to say that not all cops are bad. Yeah.

Robert Wiblin: Yeah, Sorry. I guess I don’t want to imply that. I suppose as a non-American, maybe I would come in with the assumption that the vast majority of police are trying to follow the law almost all the time. And I suppose the more I’ve seen of police behavior, the more I wonder whether they think of themselves much more than I thought as above the law and able to do what they want.

Jennifer Doleac: Right. I have to admit, I try very hard both as a researcher and as a person to see both sides and think about the pros and cons and play devil’s advocate, even in my own mind. And it has been very difficult to watch the videos of the police confrontations with protesters in recent weeks and be able to defend the police. Yeah, I think that’s part of the reason that the protests have been as large and as vigorous and have gone as they have been and have gone on as long as they have been, because people keep seeing those videos and they’re like, “This is not okay. This is not the way the police are supposed to be interacting with the public”.

Robert Wiblin: Yeah. Do we know how much misconduct is concentrated among a minority of particularly law-breaking officers, or maybe how much misconduct is concentrated in a minority of police departments or districts?

Jennifer Doleac: So we don’t. So information on officer misconduct is one of those datasets that’s really, really difficult to come by. You can sometimes get that information from individual police departments. So a number of police departments have tried in recent years to put together some predictive algorithms to figure out based on past behavior who is likely to use force or kill someone or do something really bad. That would really be a problem down the road. And most of those types of exercises are off the record and not public. But from talking with researchers, there is one paper that’s been published using data from Chicago. Everything I’ve seen and heard suggests this is very easy to do. You don’t need some fancy machine learning algorithm. Like it’s not a super difficult prediction. Basically previous complaints predict future bad behavior. And so then the question is just, do the police departments currently have the power to do anything with that information?

Jennifer Doleac: And the answer appears to be no. And that’s the sort of dismaying part that I hope is going to be addressed in the near future. There are a few options. One is we just fire you after you get a certain number of complaints and say like, “Sorry, it’s a really bad signal, and statistically you’re likely to go on to do bad things. So you’re out. You’re onto some probation period or something and you lose your job”. I think we might get to a system where we basically just expect more from our police officers. There’s this great Chris Rock sketch where he’s basically saying like, “There’s some jobs where everyone just needs to be good. Like pilots, you can’t just like occasionally crash in the mountains, right? Like you all need to be landing”. And police are probably in that category too. You can’t have some that are just out there killing people needlessly. That’s just not okay. And so maybe the answer here is, you know in a lot of jobs just firing someone because they made a couple mistakes would seem harsh, but maybe with police officers we’re okay with that. The alternative is we figure out disciplinary procedures or additional training that can actually help people course-correct. And help people get on a better track and become a better cop. And because there’s currently so little opportunity for police departments to intervene with those officers, we really haven’t had much opportunity to evaluate what works in that space.

Robert Wiblin: Yeah. I guess it seems like you’re saying that there’s some jobs where everyone has to be good at it because it’s just too disastrous if there’s a really terrible staff member. It seems to me like police, naturally, I would think, and I guess in Australia, I would hope that it’s the case that they’re held to a higher legal standard perhaps than just an ordinary person. Because they have training. It’s their job to follow the law and enforce the law. And yet it seems like in the US that they’re held to a lower standard of conduct than just a typical random person, or at least courts are so much more forgiving of the use of force or of misconduct on their part when it seems like they should just be far more strict because the police know the law.

Jennifer Doleac: Yeah. Well, so I think, again, this goes to the question of what the law is? And they are legally given more discretion to use force. Like I don’t have full discretion because it’s not my job to try to hold criminals accountable. And I think there’s this general sense that we are hiring these people and trusting them to maintain law and order and arrest people who are doing bad things. And sometimes force is going to be necessary, especially if that person are themselves violent. And so I think there are a lot of situations where it’s not going to be clear from the outside what you would have done in that situation. And so I think the spirit of the apparent laxness here is that, you know, let’s imagine we have a police force that we’ve hired and trained well and that we do fully trust.

Jennifer Doleac: Some of those people are going to be in a very dangerous job and they’re going to be in dangerous encounters and violent encounters. And in the moment, they’re going to be doing the best they can. And sometimes they’re going to be using force. And sometimes that will not work, or it will appear too harsh from the outside. So then to later on go back and say, “Well, it looks like that was too much, so we’re going to fire you” or something like that. You’re going to wind up with a lot of officers that aren’t going to want to go near anything that could possibly be violent because they’re like, “Well, I can lose my job. I’m putting my own life on the line here and I’m doing the best I can, and sometimes I’m going to think I need to use force, even if someone else might have done it differently. And if you’re not going to back me up, then why on Earth would I do this”? And so the question is just where to draw the line. And I think the problem is it’s tricky to figure out where to draw the line in terms of what is a justified use of force there. And unfortunately, in a lot of departments, it seems like they’ve drawn the line at the floor because they can’t figure out exactly where to draw the line somewhere above the clear, bad apples. And then that creates a culture where everyone, you know, I’m saying the words, bad apples, and I also don’t want to suggest this is just a couple of bad apples, because I think a lot of this is culturally pervasive. But I think that that contributes to that culture too.

Robert Wiblin: Yeah, I mean, it seems like this qualified immunity thing can go pretty far. I suppose I’ve heard of a case where a bunch of police officers stole a whole bunch of property that they were meant to collect because it was under warrant, and then the court gave them a pass under qualified immunity grounds because there wasn’t a specific role against literally stealing property to keep for themselves. Have you heard of that one?

Jennifer Doleac: I have not heard of that case. I think there are other issues around similar sorts of things that come up a lot in the civil asset forfeiture conversation where again, police have a lot of discretion to be able to seize property that has been used during the course of a crime or bought with the proceeds of a crime.

Robert Wiblin: Or that they merely suspect, right?

Jennifer Doleac: Right. They don’t have to prove it in any way. But then on top of that, the way that laws are currently written, police departments often keep the revenues that come from that. And so there’s actually an incentive to go out there and seize assets. So that’s not good. Yeah, sure does sound like theft. Yeah. I think there’s been a push for a long time for many sides to fix that.

Jennifer Doleac: I think that’s also just an example where this is actually not a difficult fix. The incentives are clearly perverse here. And so just don’t let the police departments keep that revenue. And the challenge is we’re just going to have to pay higher taxes in order to fund our police departments appropriately and fund the courts so that we don’t have all the fees and fines and civil asset forfeiture that they currently rely on to pay their bills. But that feels like a better system to me.

Global comparisons [00:42:29]

Robert Wiblin: How exceptional is the United States in terms of its criminal justice system? As I said, it’s very unusual in terms of its incarceration rate. Oddly enough, it actually has fewer police and spends less money on police officers than than many other rich countries. I guess, looking at international tables, the rates of police killings are much higher in the US than many other rich countries, but actually oddly lower than many central and South American countries. Yeah. Do you want to say anything about like putting the US in a global context, crime and policing wise?

Jennifer Doleac: Yeah. I mean, I guess this isn’t like a super rigorous comparative analysis. It”s not my forte, but we’ve had a bunch of studies coming out of especially Scandinavian countries because they have amazing data. There are all of these nice studies on the criminal justice system and what happens if you incarcerate someone and so on using the very rich data from Sweden and so on. And just recently, there has been more research that’s been coming out of Latin America. And I have been very happy about that because I think in many ways, actually those criminal justice contexts are much closer to our own than the Western Europe contexts. And honestly, I think part of it is the gun issue. We should probably expect in general that our police are going to be using more force because more people on the streets have guns. So yeah, I think the incarceration system, in a lot of ways, the Latin American context is closer to the US context and can be more informative about US policy than what’s happening in Sweden.

Robert Wiblin: Yeah. I guess naively you might think if you hadn’t looked at crime stats that much before, or at least like violent crime stats before, you might think, “Well, it’s going to be countries that are poor, that have much higher rates of violence”. But I think if you look globally, it seems like cultural factors are much bigger. That many countries in Africa are a lot poorer than countries in the Americas, but they have lower violent crime rates. I mean just the Americans as a whole really stands out, and its violent crime rate is much higher than you would otherwise expect. And then you have countries in Asia that just have phenomenally low crime rates. It’s extraordinary. It’s extraordinarily rare that people are murdered.

Jennifer Doleac: Yeah. I think a lot of that is, again, it comes down to culture, right? I mean the US is much more individualistic than a lot of Asian countries, where it brings great shame on your family if you’re convicted of a crime and like that’s not the case in the US. So yeah, I think that kind of highlights the challenge with just comparing countries and individual policies and saying, “Well, maybe it’s because your police are different that you have higher violent crime rates. Like maybe? But maybe we are just different, right? Like there’s a real possibility here that the US is just culturally different in a way that maybe we can change, but it’s going to take a long time and that’s not going to be a policy fix.

Procedural justice training [00:45:19]

Robert Wiblin: Okay. Let’s move on and talk about possible solutions or improvements. Yeah. What possible solutions to all the problems that we’ve talked about so far are you most excited about? Are there any that are worth highlighting for the audience?

Jennifer Doleac: Good question. Yeah. So there’s a new study that just came out in PNAS that has caught my attention that I find myself talking about a lot: measuring the effectiveness of procedural justice training for police officers. So basically, it’s like a one day training, this program they put together. So not super extensive. But basically really just helping police officers and pushing them to make the process of interactions with citizens feel more fair, and to leave the citizens with the feeling that they were heard and understood and they trust the person who’s making the decision. That they kind of had taken everything into account fairly. That they trust the process, basically. In general, trainings aren’t necessarily easy to measure the effectiveness of because it could be a very selected sample who takes the training, but in this case, they needed to train all 8000 plus officers in Chicago with this training.

Jennifer Doleac: They couldn’t do it all at once, so they basically had to do 25 officers at a time, once a month for many years. And they randomly assigned officers to particular times of when they took the training. So it wasn’t just like, you know, the most senior people first, or volunteers first, or something like that. And so what that leaves us with, is it’s a really nice natural experiment where you can compare the behavior of officers who took the training earlier to officers who took the training later and then measure in that way what the impact of the training is on behaviors and the behaviors that they were looking at were things like complaints received from citizens, and use of force by officers. And they found that this one day training program that officers just took once pretty dramatically reduced both complaints and use of force significantly.

Jennifer Doleac: And that was on average across all officers. And I have a hunch that if they were to zero in on just the officers that had used more force in the past, maybe they would see a bigger impact for those officers. Because you’re going to have a lot of zeros in there that kind of water it down. And it also seems like the sort of thing that repeated training could be helpful for. So I think there’s no more experimentation to be done there. But I think it’s a really nice example. Both of a program that police departments should be trying, but also that staggered rollout design that they used just because of capacity constraints, right? They couldn’t train everyone at once. That’s going to be a problem a lot of police departments are going to face in any new training program. So using a similar staggered out design where they just randomize who goes first, who goes second, who goes third, I think, is a really nice model for implementing new training programs in a way that then allows us to measure whether they’re effective. So I really liked that study. And I think that their approach is promising.

Robert Wiblin: Yeah, that’s the kind of intervention that I would expect would have no impact. It’s like, “Take people for a one day training program”. It sounds absurd.

Jennifer Doleac: So others are left really up on procedural justice for a long time, and I’ve always been so skeptical. And I have to say there’s another study, a much smaller study that had been done in Seattle with their police department on another procedural justice training program and, you know, a small sample, and like there were some issues there that just made the analysis tricky, but they also found the beneficial effects and it was like, “Well, let’s wait for a bigger scale up”. And then this study came along and it’s like, “Oh, okay. Yeah, maybe there’s something here”. I’m always happy to be wrong in that direction. That is the best case scenario.

Robert Wiblin: Completely. I mean, let’s say that that result was true and it replicates in other contexts. What does that show? That you can take officers for a day and tell them not to be terrible at their job and explain to them things that you think would have been taught during the process of training police officers to begin with. That’s very fundamental to their job. And that this can have such a big impact on their behavior. That’s why I would have been skeptical. To begin with, to get anyone to change their behavior systematically with one day training seems very hard. There’s not a great record of that. And then also you think, surely I’ve been told this before?

Jennifer Doleac: I think that the issue is that maybe they haven’t been told that before. So I think the more standard approach to policing is kind of a command and control approach where it’s like the police officer is in charge. It’s very similar to a military sort of approach. And so if you enter any sort of scene and you’re like, “Well, I’m the one in charge and you follow my orders or you’re disrespecting me and you’re out of line and then we have to do something with you”. That’s a very different way of doing your job than going in and saying like, “Hey, I want to hear everyone out. And I want to make sure everyone has a chance to have their say and here’s how I’m going to make the decision and is everyone okay with that”?

Jennifer Doleac: I mean, a day isn’t that long, but it’s long enough to not only do that, but also do some role playing and think through and practice how you would do this in different scenarios. And yeah, it appears that for many officers that’s enough to get them to try it. And then if you try it and you’re like, “Wow, that went way better than it usually does”, then you’re going to keep doing it, I would imagine. So, yeah. I think it probably says a lot about how badly the way that we usually train cops to approach these incidents goes.

Robert Wiblin: Yeah. So I haven’t had a lot of interactions with cops in the UK or Australia, but I guess I’ve heard, at least in the UK, they have a pretty different mentality than the military. I guess the term they use is “community policing”. I’m sure it’s far from perfect, but that they view themselves more as people who are meant to help and guard people and not necessarily boss them around quite so much. Yeah. Do you know of any reliable research on the different cultures of police departments in different places around the world?

Jennifer Doleac: It’s not necessarily in a sort of cross-culture comparison sort of thing. There probably is and I just don’t know — it’s not in my orbit. But this issue of community oriented policing is when that comes up a lot. I think the challenge with that and studies in the area is that everyone kind of has their own definition of what that means. Like, ideally I think the idea we have in our heads is a cop who just knows everybody and lives in the neighborhood and is super friendly. But when you think about like, “Okay, well how exactly are you training an officer to do that? Like, what are they supposed to be doing over the course of their day”? And then different places are going to have different procedures and that sort of thing.

Jennifer Doleac: So we kind of all have, in our minds, the image of the ideal police officer, but how to train people to become that officer, the approaches could differ and they do differ. So it’s kind of hard to study.

Improving hiring practices among police [00:51:47]

Robert Wiblin: Okay. We’ll ask you what possible solutions are you excited about, and you gave me an amazing answer the first time, so I’ll ask it again.

Jennifer Doleac: Right, another potential solution. So another one where there’s at least some evidence actually comes from the ’70s when a bunch of court mandates required police departments to hire more black officers and more female officers and researchers have used those court-mandated affirmative action orders as a natural experiment to measure what happened for departments that were required to make these changes. And it turns out lots of good things happened. So victimization of black residents fell apparently because black residents trusted the police more. They actually were more likely to report crimes and stuff because they trusted the police department to take them seriously. We see a very similar thing for female officers. So women are more likely to report domestic violence, for instance, when there are more female cops on the force, and you see intimate partner homicides fall. So the most serious outcome of a domestic violence case happens less often. So that suggests that the officers are actually able to do something. And so in this broader conversation about how do we… In my mind, so much of this conversation is about trust. Like how do we build trust between communities and law enforcement again, increasing the diversity of the force, again, back in the ’70s and kind of early eighties it seemed to have been successful in that way. So does that mean that increasing diversity now could be just as successful?

Jennifer Doleac:
Maybe? I think it’s obviously a different context, but it is definitely something that seems promising enough to pursue. And the hard part here is then how exactly do you go about that? And in the United States right now, most police departments are have a really hard time recruiting. You might not be surprised to find that a lot of people aren’t that excited about being a cop right now. It’s not the most prestigious or appealing job at the moment. And so when you have a limited pool of applicants, it’s really difficult to be choosy to make sure you have more of one type on the force or more of the other. And so I’ve become interested in… There are a few places that have been doing experiments with how they recruit and what kinds of messaging and, you know, flyers distributed to residents that might be targets for recruiting or so on.

Jennifer Doleac: So I think there’s a lot of work to be done. I personally think that one really helpful approach could be to find ways to make the job more appealing and draw a bigger pool of potential applicants to the jobs. That then we have the power to be choosy about who we actually hire and put on the force. And then when someone does a bad job, we have more power to fire them also because we have people to replace them. And so when we’re thinking about increasing diversity or in any other way, sort of improving who becomes a cop, figuring out ways to recruit a big pool is going to be key.

Robert Wiblin: To what extent is the police force now more representative in terms of race and gender? I assume in the past it was very white and male, but yeah, how’s it changed?

Jennifer Doleac: And I think in general it still is, but it’s better than it used to be I think is the answer. Yeah.

Increasing accountability [00:54:52]

Robert Wiblin: Yeah. Okay. Yeah. Any other policy reforms that you’re excited about?

Jennifer Doleac: What else? I mean, I think there are huge conversations to be had about increasing accountability. We sort of have been dancing around this for most of the conversation, but the role of unions here and how difficult it is for police departments to fire people who aren’t doing their job effectively. That clearly needs to change. It’s sort of amazing to me that we have any industries at this point where even if you do a terrible job, we just can’t get rid of you.

Robert Wiblin: Especially for it to be this one.

Jennifer Doleac: Yeah. Especially this one. Right.

Robert Wiblin: Why can’t it just be like if you’re a bad cook, then you don’t get fired rather than yeah…

Jennifer Doleac: Right. Well, yeah. I do think that we’ve kind of had a similar conversation in teaching and education in the United States and I think there are probably models there about how do you incentivize some teachers to be willing to sign a contract that instead of saying like, “You have complete job security and we will defend you no matter what you do. And we’re going to pay you $50,000 a year, we’ll pay you a hundred thousand dollars a year. But if you don’t do a good job, you’re fired”, and pay increases are merit pay only. So finding ways, and I’m not an education person so I’m probably dramatically oversimplifying and probably getting part of the education model wrong; that’s my general sense of the kinds of conversations that have happened around teaching. And I imagine that a similar kind of conversation in policing would be helpful, but I think there’s just a huge amount to do in that space.

Robert Wiblin: Yeah. There’s this preprint going around which is kind of claiming to show that the formation of police unions in the fifties significantly increased killings by police. Maybe something like around 10% without reducing crime. I guess potentially if that’s the case, if we can measure that, then it might’ve also increased other violent actions by police even more or police misconduct by preventing accountability. What studies do you know of that there have been of the impact of police unions? And I guess, is there any research on to what degree they’re an impediment to any of these reforms that we might want to implement?

Jennifer Doleac: Yeah. So there are three studies that are on my radar. The one that’s been published is… I think it’s been published, there’s at least a paper online by John Rappaport and some colleagues at the University of Chicago Law School, where they were looking at a court decision that allowed, I think, Sheriff’s departments to have collective bargaining rights in Florida. And so they use police departments as the comparison group because they had those rights all along or yeah, it didn’t change at the same time. And they found detrimental effects of basically being allowed to unionize for the sheriff departments. I forget exactly what outcomes they were looking at. Then there are two working papers that I actually haven’t seen the actual papers for. I don’t think they’re online yet. The one by Jamein Cunningham and Rob Gillezeau who’s been talked about a lot on Twitter.

Jennifer Doleac: Rob posted a Twitter thread that kind of talked about the findings there, but I don’t think they have a working paper online yet. If they do, I haven’t read it yet. I don’t think it’s there yet. There’s another one by Felipe Goncalves… I think he’s close to posting a working paper online. I’ve gotten a sneak peek at it, but it’s also not circulating. Yeah. So I think the Jamein and Rob paper and that the paper from Florida both find detrimental effects of unions. I think with Filipe’s, we’ll see when that one comes out, what kind of the final answers are. They might be a little different based on my sneak peek. So I think there’s still some open questions here.

Robert Wiblin: Qualitatively, I mean, we’ve got lots of good ideas here potentially, or there’s lots of papers suggesting ways that things could be reformed. It could be better and could improve police performance. But I guess my impression just from seeing what people have to say in the media and elsewhere is that police unions might not be into this. And they have an awful lot of clout including other politicians. That they might be a pretty big impediment to reform. And maybe if you don’t tackle this at the root, then we’re going to have a hard time getting up things that even maybe a vast majority of the population support if the police unions opposed to it. And I guess also, I mean, it was interesting to learn that police weren’t always allowed to unionize in the United States. So there was a time when that was not typical. When states didn’t permit that and that we saw these changes in behavior after that became permitted. So potentially that’s one thing that we need to roll back.

Jennifer Doleac: Yeah. I completely agree. This seems like the conversation that needs to be had. In my mind, it is entirely a political conversation. It’s a question of to what extent they have bargaining power relative to the cities they’re negotiating with. And to what extent their bargaining power is lower now than it used to be in the past. I suspect it’s going to be lower now than it used to be in the past.

Robert Wiblin: Why’s that?

Jennifer Doleac: Just given all of the current movement to change and the anger at police departments and police officers. There are examples of, I think in Camden, New Jersey, they had disbanded their police department and just built up a new one. And I think that seems to have been quite successful. And I think part of the reason for that is they basically essentially just broke the union, right?

Robert Wiblin: They ripped up the contract and started from scratch?

Jennifer Doleac: Yeah. They ripped up the contract and started over. And so I think other cities that wind up taking… I think that is certainly part of the conversation right now that cities might just disband their police departments and start over. And so that is one way to get around the union. And if that’s on the table now, then that alone reduces the bargaining power. Yeah. So we’ll see. I mean, I completely agree that that feels like the major constraint on most of these conversations. But I am not an expert on politics, and we don’t have studies on how to get unions… Maybe this a game theory question. I don’t know.

Overrated policy ideas [01:00:33]

Robert Wiblin: Right. Yeah. Different department. All right. So we’ve talked about a couple of positive reforms there. I want to get back to more positive policy reform ideas, but let us talk for a minute about what kind of proposed solutions do you think are out there being promoted, being circulated, that you think are shown not to work, or at least not as proven as people think, or for whatever reason are less exciting to you than maybe than to others?

Jennifer Doleac: Yeah. So one type of reform that was a really big deal in the wake of the last time we had this conversation and there was great momentum for reform and greater accountability for police officers, was increasing the use of body worn cameras. And so at this point, there have been, unlike most things in the criminal justice space, there’s actually quite a bit of research on the effectiveness of this program. And there are over a dozen randomized controlled trials at this point, testing the effects of getting cameras to some officers and not others, or everyone on a certain shift gets a camera but not other shifts as a way to measure what the impact of body worn cameras is on officer behavior. And the hope here was that if the hypothesis is that officers know what they’re doing is wrong, but they know they can get away with it, then recording everything that they do might give them pause and make them not do the things that we don’t want them doing.

Jennifer Doleac: And so we have all these randomized controlled trials now, and basically, on net, they found no effects. So in some places they seem somewhat helpful in terms of reducing complaints and reducing officer use of force. In other places, they actually make things worse and force goes up.

Robert Wiblin: I remember when I first saw those results. And I was very surprised. I would have put money on it helping. On it working.

Jennifer Doleac: Yeah, and the first big study like this in the United States was done in Washington DC, and they just found no effect on anything. And so there are a couple of possible reasons for this, right? Let’s say one is, especially in the current context, every cop assumes they might be on camera all the time anyway, because everyone’s got cell phones and there are cameras everywhere, businesses and whatever else. And so they were already treated, to the extent of having their behavior filmed.

Jennifer Doleac: Another possibility is the hypothesis about the officer’s decision-making in that moment is wrong. It’s not necessarily that they’re like, “Well, I know I don’t need to use force here, but like, I want to anyway, and nobody’s gonna notice”. It might be that most force is used because police are legitimately afraid and that could be a training issue, right? Like maybe they shouldn’t be afraid. And there are other ways we could help them figure out how to deal with that situation. But if they’re using force because they’re legitimately afraid for their lives, then putting a camera on them isn’t going to change what they do. They think they’re doing the right thing in the moment. The other possibility is that we now have all this footage, but because of the fact that we’ve been having this conversation about how we can’t fire anybody, there really isn’t much accountability.

Jennifer Doleac: We’ve got the footage and it might wind up being leaked to the press maybe. Honestly, it’s probably more likely that a cell phone video will go viral and be released than body worn camera footage. But if ultimately nothing’s really going to happen to you, except that you’re in the news for a few days, then it maybe doesn’t provide a strong incentive to be on better behavior than we would like.

Robert Wiblin: Yeah. Interesting. I think I saw you recently tweet a paper that suggested a more positive result from body cams?

Jennifer Doleac: There was. So I still haven’t had a chance to read it closely, but there was another RCT in Las Vegas which seemed to find beneficial effects. So add that to the list. But I think at the very least, we can say this is not going to solve the problem.

Jennifer Doleac: I think a lot of police departments really felt pressured to do something the last time we had this conversation. And of course there were some private firms that were happy to step up and sell them a product. And body worn cameras themselves are cheap. It’s the storage cost for all the video that’s extremely expensive. So you’re paying a lot of money for this. And that can be fine. We can just want it in case we need the footage for some sort of case down the road. But if the hope was that if we’re spending all this money because we think it’ll change officer behavior, it does not seem to be a slam dunk on that outcome.

Activists and causal claims [01:04:39]

Robert Wiblin: Yeah. There’s a police reform organization that many people might’ve heard of called “8 Can’t Wait” which has been promoting an 8 point platform of stuff that sounds pretty sensible to me like banning chokeholds and strangleholds, requiring police to engage in deescalation, requiring warning before shooting, requiring comprehensive reporting of use of force. I guess, I think I saw you on Twitter suggesting that perhaps they might be overstating the degree to which there’s strong evidence that these approaches will actually work. Is that right? And maybe, do you have any intuitions about whether they would work?

Jennifer Doleac: So I agree that most of those moves seem totally sensible. I think honestly that’s the way they should have sold this whole agenda, like say “Here’s some great ideas — did you know that some places allow officers to do this? Maybe that’s a bad thing and we should stop them”. The problem is that they sold it as evidence-based data-driven. They basically said like data… I believe the phrase was “Data proves that these eight changes will reduce police killings by 70% or something”. And people like me of course see something like that and I’m like, “Wait, where’s the study? Like, that’s amazing”. So I look at the study and I don’t want to undersell the tremendous work this group did. I mean, they managed to catalog a whole bunch of policies across cities and states and that’s really hard work.

Jennifer Doleac: And so that was a tremendous service. But then they just did a kind of raw correlation between, “Well, let’s look at places that have banned chokeholds compared to places that don’t”, and it’s like, “Oh, the places that don’t, have fewer killings”. And so then they attributed that difference to the difference in policy. But if you think about, you know, there are likely other underlying differences between those places. And so I imagine sometimes you’re like comparing San Francisco with rural Alabama and, you know, maybe it’s the chokehold ban that made all the difference, or maybe there are other differences between those two places. And so researchers, you do not need an RCT to be able to measure causal effects, but you do need something better than that. And so as a researcher, I recognize the appeal of being able to say that your solution is evidence based and data-driven. Unfortunately, we see that those words used so often in context where I think the evidence is quite weak and it gets to the point where those words don’t mean anything anymore. And so that is something I pushed back on strongly. If your case for these policy changes is just they’re good ideas, then make that case, but don’t pretend it’s evidence driven. Because that claim is false. And so I said that. Other researchers have said that. I think one of the people involved in 8 Can’t Wait actually just resigned earlier this week. I mean, they have stepped back.

Robert Wiblin: Over this issue?

Jennifer Doleac: I think it’s just a cautionary… I mean, I don’t wish harm on them, but I guess I am somewhat gratified that people did push back and said, “Actually it’s not okay to oversell evidence”. I hope we hold people to that standard more often. So yeah, I think it’s a cautionary tale of about overselling evidence as the basis for whatever reform you’re proposing. Like, this a space where we basically have no evidence. We know so little about what’s going to work here. And so most of the stuff that we’re going to want to try, we’re just going to want to try, because it seems like a good idea and that’s going to have to be enough and that’s fine. And so go do things that seem like a good idea. But my pitch then would be also be humble about the likelihood that that’s probably going to fail. So just be prepared to evaluate and change course if it doesn’t work. So I pushed for the evaluation and evidence after the fact, I think, pretending that we have the answers now, or that we should be basing all policies on evidence in a space like this, it’s just impossible. We can’t.

Robert Wiblin: Yeah. I guess I feel like I want to stick up for 8 Can’t Wait a bit.

Jennifer Doleac: Go for it.

Robert Wiblin: Well, I suppose one thing, I mean, I’ve been following Samuel Sinyangwe who I also invited to come on the show. Unfortunately he’s, I think, swamped with other work right now. So maybe one day. Yeah, what can I say? Well, I suppose one thing is even doing those crosswise comparisons is a lot better than a lot of other social science that I see. It’s almost like a higher standard of evidence, even though I guess it doesn’t really bring you causal identification.

Jennifer Doleac: You’re reading the wrong studies (laughter).

Robert Wiblin: Maybe I’m reading older studies as well. I guess I worry about doing this myself. It’s easy to get up on your high horse about accuracy and precision and kind of technocratic stuff like this when it seems like taking that approach maybe just doesn’t galvanize people in the same way. It doesn’t bring people out to the streets. Perhaps it’s harder to build a movement. You know, it’s very important to have, I guess, policies that you know work, but then it’s also very important to be able to get things implemented. And perhaps campaigners, activists, they kind of bring a different skillset, a different approach, a different mentality to things. And I agree that they shouldn’t oversell that way, but I think, and I guess I see this with whenever anything becomes a mass movement, the evidence and the way that people talk about things falls below the standards of an economics PhD seminar. People start becoming a lot more lax. But I think that that’s just an inevitable step in the path of a movement becoming massive which, in this case, it seems like that’s something that’s going to have to happen in order for any of these things to actually happen.

Robert Wiblin: It’s not enough to just have people who are very into the social science and will read the papers to be excited about these things. I suppose I’m not sure exactly where I’m going…

Jennifer Doleac: Look, I mean, I spend a lot of time working with policy folks. I’ve been working with the Council on Criminal Justice, and I’m usually the only researcher in the conversation. And so I spent a lot of time talking to people who are like, “Yeah, your fancy studies with identification strategies are nice and all, but it’s just not gonna be part of the conversation most of the time”. And so I’m aware that that is an exceedingly high bar and needs to be adjusted. I think I would rather that organizations just be very transparent about what they can and can’t show. So in this case, I think it would’ve been totally fine if they’d said, “Did you know that places that have chokehold bans have this rate of bad behavior and places without chokehold bans have this rate”? Just tell us what the data says.

Jennifer Doleac: It was the very strong statement of the causal effect that made me upset, but also made a whole lot of non-economists upset. And I think partly because it was just so easy to see through. It didn’t take a PhD in economics to see that they didn’t actually prove this. And I guess the reason I get riled up about this, isn’t just that I’m an economist. This is my bread and butter. This is what I do all day. But I do think that there’s real danger in overselling research and saying the research proves X because we’re living in a time when I think expertise is often undervalued and ignored. And I think the danger here is that you say like the research shows this will work.

Jennifer Doleac: The research shows “Implement these eight policies and deaths will fall by 70%”. You go do those things, and if it doesn’t work, then everyone’s like, “Oh, research is useless. I told you we shouldn’t be listening to researchers”.

Robert Wiblin: Or, this problem can’t be solved.

Jennifer Doleac: Or this problem can’t be solved. Right. We should just all go home. Because we tried. We tried the best evidence-based approach. The data proved it. Yeah. So I would rather us all just be very transparent about what we are showing and what we’re not and frankly call out the advocates when it is. I totally agree. This is a thing that advocates do, but I guess what I’m saying is they shouldn’t be doing it and we should call them out on it when they do it. And I think if we care about finding solutions that work, then we should be pushing for evidence-based solutions, but we should insist that that term means something.

Robert Wiblin: Yeah. I’m still trying to figure out what my views are on this, but I feel like there’s something where I’m like not agreeing with, I guess a lot of people who I follow on Twitter, or a lot of my friends. Maybe I first started to have this feeling with the Extinction Rebellion climate change stuff last year, where I saw a lot of people who would point out that things that are being said at these rallies or things that were being said by activists that were not quite right, I guess exaggerations of how bad climate change might be or proposing policies that aren’t the best or aren’t going to be the most cost-effective. And I guess I wanted to stick up because I’m like these people have managed to build a huge movement to solve this problem that you think is really bad and it’s just not possible, I think, to get so many people out on the streets and to build such a broad based movement without the intellectual standards going down. And you’re always going to be able to say, “Once you get these big protests, once you get so many people newly interested in something, people who don’t have PhDs. Maybe didn’t go to university. Haven’t thought about the social science. Haven’t thought about the cost-effectiveness of different technologies”. Necessarily, you’re always going to be able to say “Look at all these silly things that they’re saying”. And I guess I feel like I want to kind of steelman the position that they’re making and say that there kind of has to be a place inasmuch as we need a broad-based movement in order to get policy reform, then we just have to accept that there’s going to be a stage that we go through where there’s lots of dumb things being said. Lots of false things being said.

Jennifer Doleac: Yeah. But this wasn’t a situation where someone had this written on a poster board that they were holding up. Like this was the leader of this organization. So one of my favorite threads about this was by Eve Ewing, where she was like, you know, somewhere late in the thread, it was like every thread was just this scathing… I was very thankful to her, especially because she’s not an economist and she was also offended by this. But she had a couple of tweets about this like “They were in a position to know better”. I think the flip side of what you’re saying… In some way, when you build a broad-based movement, yeah, you’re going to get more of those people out there holding up the signs that might not be quite right.

Jennifer Doleac: But it also means that the people who are leading the movement have resources and have volunteers with skills to know. And so in this case, I think Eve said something like, “You just got all of this data. You have this data for years and you’ve gotten all this money. You should have used that money to hire an RA who knows better”. And so I think that’s the point where I completely agree that building movements is an incredible skill and I do not want to sell that short at all, but I do think it’s fair to expect the people at the top to–

Robert Wiblin: To being more careful about what they’re putting out.

Jennifer Doleac: A little more thoughtful about the facts that they’re putting out about stuff like this. And I think, yeah. I mean ultimately, I think this is just about being honest about what your case is. Like the best case for the reforms they are pushing is not the evidence. It’s something else. So make a sincere case based on why you’re actually pursuing these things. And I guess I would like to think that that would be effective.

Robert Wiblin: Okay. Let’s push on and think about other solutions that you may be more excited about that I think there’s stronger evidence on. You talked earlier about ending drug prohibition, reducing length of prison sentences; I guess one thing about ending drug prohibition might be that there would then be less interactions between police and the public or less harassment, less searching people for drugs. Less dealing with people who might not be super keen on the police and things might escalate, even though there was initially no serious crime committed. Do you have any thoughts on whether that could have a big impact?

Jennifer Doleac: Yeah, it might. I think ultimately… I actually have a student who’s working on this issue. I know some others who are. I think there are reasons that it might have the benefits that you’re describing and there are reasons it might do nothing. In the sense that if officers are just looking for something to get this kid off the street, then they’re going to charge you with whatever. And if it’s not this–

Robert Wiblin: They’ll hassle them for jaywalking.

Jennifer Doleac: Yeah, exactly. It’ll just be something else. And so I think there are people who are kind of using some policy changes in this area as a natural experiment to see what actually happens to arrests and charges and that sort of thing to see if it actually does help people and who it helps. It might not be equally distributed across the population.

Robert Wiblin: Yeah. Yeah, you mentioned earlier this option of just disbanding a police department and rebuilding it from scratch which is something that a lot of people are talking about just now. I guess, intuitively to me, it kind of makes sense that once an organization reaches a particular level of dysfunction or of having a bad culture, maybe the only way that you can make it work good again is to just get rid of it and start rehiring people from the ground up. There’s this instance in Camden, in New Jersey. Do you know of any other evidence or experience, or common sense on that question of whether that’s a good way to go?

Jennifer Doleac: I don’t. I mean, I think the reason Camden has gotten so much attention is they are probably the only example of it happening in the past. I mean, this is where we run into issues of like it’s only possible that evidence on the effectiveness of stuff that’s actually been tried before. And even like one city doing it, you know, it’s basically like at best an event study, right? There’s no way to have a control group then. And so, yeah. I mean, we’re kind of venturing out into the unknown here. Maybe there are examples from other policy spaces. I mean maybe like schools, again, it’s another place where unions have sometimes been resistant to change. So that’s why I keep going back to them as a potential model here. But I think there’s just a lot we don’t know yet about what’s going to be most effective.

Should there be more, or fewer police officers? [01:17:34]

Robert Wiblin: Yeah. Okay. How about replacing police with non-police responders? That’s something that quite a lot of people are talking about saying, “Police aren’t really suitable for handling people with mental health problems necessarily, people who are just taking drugs”. Bringing in someone whose mentality is one of, kind of command and control, and carrying a gun maybe is a bad idea. So instead we need to have more social workers or health workers who should be answering these calls instead. What do you think of that?

Jennifer Doleac: I think it’s another thing that would be really interesting to try, I think. Yeah. So when I was working with the lab at DC, they were actually doing an interesting experiment that I think I just looked up the other day and I think the experiment’s concluded, but they haven’t put the results out yet where they were actually… It’s a slightly different context, but they were trying to reduce the number of ambulances that they sent out for medical calls. That the person did not need to go to the ER. Like it was actually not an emergency. And so what they were experimenting with was sort of a parallel kind of context there, right? Like in this case, reserve the police for really serious real crime situations versus just send them out for every call. And so in that case, they reserved the ambulances for real emergencies.

Jennifer Doleac: And so what they did was they set up sort of a triage system where the dispatcher taking the call, once it was designated a medical call, they would ask them a series of questions. And then if it seemed like they need an ambulance, they’d him an ambulance. But if it didn’t, they would connect them with a nurse that was kind of waiting: an alternate dispatch system so they could talk through whatever it was with the nurse and maybe the nurse would arrange an Uber or something to an urgent care clinic instead of the emergency room. And so because there were capacity constraints initially, they weren’t quite sure. This wasn’t something where they could just switch over overnight to this new thing. They only had so many nurses and so on. So it actually allowed them to test what the effectiveness was.

Jennifer Doleac: So they set up a system where every medical call that came in for 911, they randomized to either business as usual and you just get an ambulance, or you go through this triage process and they ask the questions, and then you either get an ambulance or get sent to the nurse. And so the results we’re waiting for are like what the impact was. Like does this lead to better health outcomes and so on for the folks who didn’t need the ambulance in the first place? And so you could imagine setting up something similar in this policing context where the calls coming in for police, you randomly send some just the police, but then as we’re building up our capacity to send social workers or whoever else instead, you could kind of have this randomization where you ask them questions and then you’re like, “Well, all right, yeah, you need a police officer” or, “You know what, we’re going to send you someone who could take better care of what you actually need in this moment”, or connect them on the phone or they come visit in person or whatever. So I’m kind of hoping that the system that DC has set up will be a good model for places to test this sort of thing. I do think that police officers would tell you that a lot of the calls they go on, they do not want to be going on. There are mental health calls or parents in an argument with their children or just crazy stuff that there’s no reason to call a police officer here, but people don’t know who else to call. There’s no one else to call. And so if there were someone else to call, could that make us all better off? So I think that’s a really useful conversation to have.

Robert Wiblin: Yeah. I put this idea of having non-police community respondents to a friend of mine who is extremely critical of the criminal justice system in the US in general, but they actually just spoke up in favor of the police saying there’s good evidence that the existence of police, that the number of police does reduce crime and crime is a really big problem in the US in general. And it’s a particularly big problem for people of color, especially like to be victimized by crime. And they were worried that if all we did was reduce police numbers and then we replaced them with social workers, that that could end up increasing crime on net and could be a bad thing. I guess it’s a lot more biased to speak up in favor of the police. But my understanding is that there is good evidence that police do reduce crime. Do you have anything to say on that?

Jennifer Doleac: Yeah, absolutely. So I think this is gonna be part of the nuance in this conversation that we’re going to have going forward. There is thin evidence on a whole bunch of stuff. But one thing we know at this point is that hiring more police officers reduces crime. Now the conversation we’re having is “Yeah, but they might also increase a variety of social costs to the communities that they’re policing”. So they might reduce crime, but the negative interactions that people have with police while they’re doing their jobs might carry such large costs to those individuals that that’s something that could cancel out some of the crime reduction benefits we’re getting. Yeah. And so like a socially optimal system might actually be one where we have higher crime rates, but fewer of these other social costs.

Jennifer Doleac: And so that’s something. Economists love to put numbers on things. And we’re very focused on measuring causal effects. And there are very few studies at this point that really do a good job of measuring the causal effects of policing on, say, educational outcomes or emotional trauma for kids in the neighborhood or something like that. So we kind of have a sense that like this is a big social cost, but we’re not really sure how big it is and then how to compare it to the crime reduction benefits. So I think in this conversation about like, could we shift some of the responsibilities from the police officers to other people? I think the hope is that if we could have the police do the real crime fighting stuff, but have everyone else such as the social workers do the other stuff, then you can basically reduce a lot of the unnecessary negative interactions which carry costs, but keep all the crime reduction benefits.

Jennifer Doleac: I think that’s the hope. I’m talking with lots of policy folks these days, and people who are actually in places where they’re trying to make these decisions about what to do. What they’re worried about is there’s all this push, you know, the call to defund the police, which turns out like the definition of that is very complicated. But the label, on its face, it’s like “Cut the police budget”. And so what they’re worried about is basically the city council is just going to cut the police budget and everyone’s going to go home and feel like they solved the problem, which is not what reformers are pushing for to be clear. But I think there’s always the danger that the most simplistic versions of the mottoes or the slogans wind up being what’s enacted and then it doesn’t fix anything. So I think in that case, if we were to just cut police budgets, and there’s a very real chance that we would just see crime rates go up. And so yeah, your friend is right to be raising that.

Robert Wiblin: Is there any stylized fact of increasing the number of police by 10% reduces crime by some amount?

Jennifer Doleac: My favorite go-to paper on this is by Aaron Chalfin and Justin McCrary. So in general, the economists measure this in terms of elasticity. So it’s sort of like a percent change for a 1% change. So if you increase the number of police officers by 1%, we can generally expect somewhere between a 0.1 and a 1% reduction in crime. So that’s a pretty big range, but it’s negative. And it’s pretty big. I think the way to think about this is just in general, hiring more police officers is one of the most cost-effective ways to reduce crime. And if we were just looking at the cost of officers and the social cost of crime, then our cities are massively underpoliced, not overpoliced. Like it would be cost-effective to add more officers. And it’s really only once you add kind of the additional social costs that we haven’t done a very good job quantifying them yet, that you get to a point where it’s like, “Oh, maybe that’s not right and we should be cutting police forces”, but that’s fuzzier in all our minds.

Robert Wiblin: It’s not obvious.

Jennifer Doleac: Yeah.

Robert Wiblin: Yeah. Let’s dwell on this for a minute. I would not have guessed that police would have such a large effect. I suppose… Well one thing, looking at the broader picture, it seems like police in the US are not super accountable. You might not expect them to be doing a very good job if they’re doing all these other bad things. It doesn’t seem like they’re potentially hiring the best people. I guess also just you put more police out on the streets, most crimes just go unsolved anyway. People consistently get away with theft. It’s not that hard to commit a crime and get away with it in reality, even with more police. So to me, it’s a bit odd what is the causal mechanism that’s causing this? That’s allowing police to be so effective at preventing crime?

Jennifer Doleac: Yeah. So I think that the general consensus is that it’s mostly a deterrent effect. So it’s not necessarily like catching people after the fact and locking them up, although there is a crime reduction effect from incapacitation as well. Like if you put someone in prison, they can’t commit more crime out in the street. But most of the effect, I think, in most studies that are able to figure out what the channel is, it’s coming from deterrence. So, you know, there are all of these randomized controlled trials of like hotspot policing, where we put a cop on this corner, but not on this other corner over here. And we randomized where we put them. And those studies routinely find a big drop in crime on the street corner where you put the cop.

Robert Wiblin: So what’s going on is maybe prospective criminals just see that there’s lots of police around and then they decide to commit fewer crimes.

Jennifer Doleac: Yeah. I think a lot of crime is just like crime of opportunity. You know, it’s not carefully premeditated. It’s like, “Oh, I see a laptop left on that park bench. That person doesn’t seem to be around. Maybe I’ll grab it”. And if there’s a police officer standing right there, you don’t do it. And so, yeah, I think that’s basically the story.

Robert Wiblin: It’s interesting. I heard someone facetiously say, “Well, because of this, we should just hire lots of police officers and get them to stand around on street corners and kind of do nothing so they won’t harass anyone”!

Jennifer Doleac: Well, so cities do this. Like I did a ride along in Chicago once and the officers I was riding along with were like, “People hate getting stationed in that kind of stuff because there are people who are just sitting in their car on a street corner and they’ve just been stationed to be sitting in their car all night just to sit there. And it’s really boring”. But I think this comes from the sense that that’s actually one of the most cost-effective ways to reduce crime right on that street corner.

Robert Wiblin: Yeah. They sound kind of like scarecrows, I suppose, just sitting there not doing anything.

Jennifer Doleac: Totally. Yeah. So you could imagine if it’s super effective to have someone there, they don’t necessarily need to be a trained police officer, although maybe it wouldn’t be as effective if the person didn’t have a gun and didn’t have the ability to come and rescue and stuff. So Jane Jacobs who was this sociologist, I guess, urban planner or something like that, who had this theory about cities — like the safety of cities really depends on how many people are just out on the streets. So sort of a very similar story where we expect that there are lots of people around who could witness you doing bad things, you’re less likely to do them because you’re more likely to get caught. And so the people around don’t have to be police officers, but kind of, depending on the situation, it might help if they’re a police officer, I guess. You know, you’d be more sure that they will intervene if they’re police officers and they have more authority if you were to resist. But yeah, I think just in general, knowing that people could see you doing that stuff can have a big deterrent effect.

Robert Wiblin: Yeah. I guess the fact that police have a big effect on crime rates makes it even more important, more urgent to get up the standards of police interactions with the community. Because, I mean, we can’t have lots of police. There won’t be support for that. If it’s the case that they harass or assault people at a level that just means that people think that they don’t trust the police. They indeed fear the police. They won’t call the police.

Jennifer Doleac: Yeah. They won’t cooperate with investigations. So in going back to your point about clearance rates, like a lot of crimes go unsolved, including a lot of murders go unsolved. I think part of the explanation for that is probably that a lot of the communities that are often victims of crimes, as you said, don’t trust the police enough to talk to them and help them find the person who did it. And so if you don’t have the cooperation of the community, you cannot do your job as a police officer.

Robert Wiblin: Yeah. I suppose there’s a whole other side of this that I haven’t really seen quantified that much, like each active police misconduct has this direct cost, but then it may also just have this huge kind of hidden indirect effect where people don’t talk to the police and they don’t support having police. And then that leads to more crime. And I guess it could be one of the drivers of why crime rates are higher in the United States.

Jennifer Doleac: Yep. I think that’s right.

Robert Wiblin: Yeah. Possibly this might also explain why I was surprised to learn that, in fact, countries in Europe spend more on police and have more police than the United States because in almost any other way, the United States justice system is much more overbuilt and extravagant and the punishments are more severe and there’s more people in prison and so on. But on this one thing where it’s like maybe you could prevent the crimes from happening in the first place, there’s a bit less on that.

Jennifer Doleac: Yeah, I’m not sure. I hadn’t actually seen those numbers. I’m not sure what… Maybe it’s some sort of returns to scale or something, although all the police departments are really tiny, you know? I mean it’s all totally decentralized so I’m not sure.

Robert Wiblin: That’s another one. So the US has 18,000 police departments I read. That seems odd. And I guess also it means that you would expect far greater variety in the quality. That it could be that there’s some really outstanding police departments in the United States, but it’s like when you have 18,000 of them, there’s going to be a bunch of just terrible ones you would expect, unless there’s a huge amount of quality control where they’re being shut down if they’re no good, which I guess doesn’t really happen. Do you think it would improve the average quality of police if, say, they operated at the city level rather than the county level or at the state level rather than the city or county level?

Jennifer Doleac: That’s a good question. So in general, I think economists probably more than anyone else err toward having things be decentralized and operate as locally as possible because then it helps… You’re closest to the community you’re trying to serve and the preferences of the community you’re trying to serve. That said, if you’re building a system from scratch, we might imagine doing it in a way that allowed more oversight and control from a higher level. I mean in practice, the way that control is operated is through budgets. So the federal government gives out a lot of grants and stuff. And so basically you can condition that funding on following certain rules. The DOJ has a lot of power to intervene and do investigations if a police department is just out of control.

Jennifer Doleac: And so you had these consent decrees where the federal government is actually very heavily involved for a little while. So there is, I mean, there is oversight, but there is, you’re totally right, there’s tremendous variation. There’s some wonderful police departments. There’s a real problem with some police departments. But I think the upside of this is going to be that in this current moment where we’re like, “Gosh, we have no idea what to do”. If all 18,000 police departments do different things, we can learn really fast what works, right? And actually my biggest fear is often what you see is everybody sort of following what their neighbors are doing just because no one really knows what to do. And so they don’t want to do something different because there’s safety in the herd. If that fails, they won’t look like an idiot because everyone else was trying it too. And yeah, we should be going in the opposite direction right now.

Robert Wiblin: On the question of like what level of government should things be run at? I mean, there’s a lot to be said for having things be local. But one thing I worry about is when things get too local, people just don’t have the bandwidth to be thinking about their local school and really holding it accountable and their local police department holding it accountable and like local utilities and these other things, you know, voters and politically active people only have so much attention to go around. And I worry that when there’s so many different bodies… People pay a lot of attention to national politics and getting more and more nationalized. And I guess it means that there might be just neglect of whether anyone is really paying attention to these really local organizations that then can kind of run out of control because there’s not enough people watching and complaining.

Jennifer Doleac: Yeah. Potentially. So in the US, education is all local too. I mean, just like all this stuff is local. I generally think of it as being a huge benefit in this space because I mean the worst-case scenario in my mind would be that everything is controlled by the federal government because our federal government is doing nothing right now. It’s very difficult to get changes through Congress, but like things happen at the state and local local level. Like things get done. And so that makes me more confident that reforms will happen because it’s not going to be as politically contentious, I think, partly because of what you’re saying. It’s not going to be quite the political firestorm or focus of attention of all the cable news shows because it would have to be a separate one in every locality. And so you can kind of avoid the political stalemate that results at the federal level.

Policies to try to reduce racial discrimination [01:33:52]

Robert Wiblin: Yeah. Okay. Yeah. One class of reforms that we haven’t talked about is policies to try to reduce racial discrimination in particular. We’ve talked mostly about ways of just improving things across the board. Are there any policies that we know of that can cause police to treat people of different races more equally?

Jennifer Doleac: Hmm, in the policing context I don’t know of any studies off the top of my head. I’ll probably remember some as soon as we get off the phone. But in general, I mean, what we were talking about earlier about police discretion I think points to me that often implementing really strict rules that limits individual discretion can be beneficial. So this is like that Jeremy West paper where it’s like when they’re figuring out whether or not to write you up for the expired registration, they give the white drivers a pass, but not the black drivers. So then that discretion is generally exercised in the sense of like, in that one direction you’re lenient on the people that are like you, but you don’t use the discretion for good. For the people that you’re less sympathetic to. And so to the extent that you just don’t have the options. Like let’s say you have to take a photo of the registration and it automatically reads the date or something.

Jennifer Doleac: Then there’s no way for you to decide whether or not to write them up. There’s a paper in the court context, looking at federal sentencing guidelines, I think, and basically showing that the end of federal sentencing guidelines wound up increasing racial disparities in sentencing. And it’s basically the same sort of story. When you give judges really strict guidelines about what the sentence can be, they have to do it. But then as soon as you remove that, they use their discretion and they apply their discretion in a way that inevitably allows racial bias to seep in. And so I think there’s this general sense… I often describe it as like a pendulum swinging back and forth within the US, conversation about removing or allowing discretion. And this comes up most with judges. You know, we like the idea of people being able to use their discretion for good.

Jennifer Doleac: We like the idea of, you know, “The strict rule said you should get 20 years in prison”, but the wise judge can look at you and see that clearly the numbers that went into that calculation weren’t representative in your case, and you had some special circumstance, and so you want that wise judge to be able to solve that problem. But then the pendulum swings the other way, and then we see cases of the wise judge gives the white person two years and the black person 20 years. And we’re like, “Ah, shrink the disparity. Like take the leeway away from the judge”. So I think what people want is for people be able to use there discretion only for good. But discretion comes with both the good and the bad and human beings, in general, are biased. So whenever we allow humans to make decisions where there’s limited course correction or limited oversight, we should expect racial disparities to emerge. So I think finding ways to reduce that kind of discretion feels like a promising avenue to me.

Robert Wiblin: Yeah. Surely there’s been attempts to create training to try to reduce differences in treatment of different races? Do you know of any research into whether that works? I suppose I’m a bit intuitively pessimistic.

Jennifer Doleac: Well, so I mean the most popular training at this point is implicit bias training. And that certainly had the goal to reduce racial bias and behavior. My understanding is that there is little, if any research supporting the effectiveness of implicit bias training and certainly most psychologists that I know think it is total garbage, but it sounds good. This is another place where it’s like, we could have just admitted to ourselves that… There’s an incident in the US a while back where there’s a Starbucks employee that kicked two black customers out. And everyone’s like, “This is crazy. We need to do something about this”. And so Starbucks shut down for a day or two to have implicit bias training and everyone else adopted implicit bias training and it was like, “This is an amazing opportunity to figure out if implicit bias training works”, just implemented it in a way, again, the staggered roll out could be helpful if you just do this right.

Jennifer Doleac: We can actually figure out how to solve the problem. And, you know, the cynical side of me is like, “Big corporations don’t necessarily want to solve the problem. They just want to win the PR battle”. They just want to look good. And so as long as everyone thinks that this will work, that’s all that they need. But yeah, it’s another place where I think it just drives home the point that we can do whatever it is we’re going to do, but just acknowledge we don’t actually know yet if it’s going to be a solution. And if we’re really serious of finding a solution, we should be willing to evaluate it and then do something else if it doesn’t.

Robert Wiblin: If I wanted to speculate, maybe not enough conversations have been about how do we avoid hiring racist cops or how do we fire the ones who show racist attitudes? I mean, it does just seem like a remarkable number of people express just… It’s not implicit racism, it’s explicitly racist attitudes. And I guess I don’t know how much is done to filter those people out so they can never be on the streets with a gun.

Jennifer Doleac: Yeah. Yeah. I’m with you. I think this goes back to the conversation we were having about hiring and what does it mean to be a good cop and how can we tell what is the line of questioning during the interview that can help you hire the people who are less biased rather than people who are more biased and how do you identify when they’re on the job who’s being biased and who isn’t? It’s not like there’s a right answer from case-to-case about what should have happened, and then you can see like, “Well, he gets it wrong for all the black residents”. There’s not like an answer key we can compare you to. And so a lot of times we can say something about averages across people or something like that, but being able to identify bias by an individual officer is harder unless you have something akin to a natural experiment which is often tricky.

The effects of DNA databases [01:39:48]

Robert Wiblin: Let’s push on. Two of your more famous results found that ban the box policies, which were designed to help people convicted of crimes get jobs didn’t have that effect at all. And actually substantially increased discrimination against especially African American men with a high school degree or less. And then you have another one which is not about crime, but found that the same product listed by hypothetical black sellers on a second hand goods market, I guess like Craigslist, got 18% fewer offers a purchase from interested buyers and the offers they did receive were 11% lower than those listed by hypothetical white users. But actually I’m especially keen to talk for awhile about ways to prevent crime that don’t involve prisons or don’t involve police. Simply, there’s a lot of low hanging fruit that there might be of ways that we can just stop crimes from happening in the first place, or stop people from being interested in committing crimes before it happens. I don’t know, like Minority Report, but maybe using light bulbs instead of amazing technology.

Jennifer Doleac: Yeah.

Robert Wiblin: So yeah, on that topic, you have this amazing paper which I was really stunned by called “The effects of DNA databases on the deterrence and detection of offenders”, which I guess changed my mind on the issue of DNA databases which I think previously I would have had a negative vibe towards. Maybe just to quote a report on this paper, “In a study of more than 38,000 males who were arrested for crimes roughly equivalent in severity to felonies in United States, the research team found that being added to a DNA database reduced re-offending by a stunning 43% and the harsher criminal sentences that are favored by self-styled tough-on-crime politicians have shown no evidence of producing any comparable level of crime reduction. And Jennifer’s US work further notes that deterring a crime by a DNA database has cost less than 10% of the cost of producing the same deterrent effect by building more prisons”, I suppose, maybe that might be because prisons don’t really deter criminals that well at all. Yeah, what methods did you use to establish that effect and what implications do you think that has?

Jennifer Doleac: Sure. So, yeah. So I have worked on the effectiveness of DNA databases, both in the US and in Denmark. So that whopping 43% number is coming from Denmark where we had amazing data. So DNA databases are basically computer databases that contain an identifying number essentially, that identifies you based on your DNA profile. And it depends on state law in the United States and national law in other places, which categories of offenders are required to provide DNA to the database. So typically you kind of start with the murderers and the rapists at like another time. And then a year or two later, you might add anyone convicted of robbery and so on and so forth. And so in both the US and Denmark context, there was an existing database, but then at some point they expanded the database to include an additional category of people.

Jennifer Doleac: And so what you can do then that sets up a situation where let’s say the expansion goes into effect on June 1st, you have anyone who was, say, charged with the crime on May 30th doesn’t go on the database. And anyone charged with the same crime on June 2nd does go in the database. But there’s nothing else that happens discontinuously at that June 1st threshold. And so we would expect those two offenders to be very similar. You know, anyone who kind of looks observationally equivalent is probably equivalent in terms of their likelihood of re-offending before the database came online. And so what we can do then is compare those people who are like just on either side of that start date over time and see what happens to recidivism. And in both the US and in Denmark, I find big drops in recidivism. And so in the Danish context, again, we have much better data so we can measure it much more precisely and you’re able to kind of separate out the effect.

Jennifer Doleac: The goal here is that if your profile’s in the database, it can be frequently compared with crime scene evidence. And so the reason we might expect a deterrent effect… Should have said this at the beginning, is that you know you’ll be more likely to get caught if you offend again. So basically if the database matches you with crime scene evidence, then that’s a lead that police might not have had otherwise. So it’ll identify you as a suspect in a crime if you might not otherwise have been a suspect. So yeah. So we see this big deterrent effect, but the DNA database itself could actually lead to an increase in likelihood that you get caught. I think that’s the whole point. And so if you just sort of look at the data we might’ve expected, if it had no effect on recidivism, we would expect an increase in the likelihood of being reconvicted, right, because you’re just more likely to get caught.

Jennifer Doleac: And so the fact that, especially in the US context, where I had to just look at the net of those two things, I still see a significant negative effect means that that’s an underestimate of the benefit. So I find that in the United States, being required to submit a DNA sample to the database reduced the likelihood of a new conviction within five years by about 17% for serious violent offenders. And that was statistically significant. It reduced the likelihood of another conviction by 6% for serious property offenders. So a little bit smaller. And then, yeah, in Denmark, it was like a huge expansion of the database in 2005, I think. And so it was essentially going from adding only sort of the most serious offenders in only a few cases to adding anyone who is charged with the equivalent of a felony.

Jennifer Doleac: And so yeah, just found huge effects on their behavior for that very broad group. So I think my prior going into this line of research was probably similar to yours. I was like, “Well, this’ll probably have benefits in terms of catching offenders more quickly”, but I think I was skeptical.

Robert Wiblin: It’s seems like it’s psyching them out. I guess they see that they’ve gone into a DNA database and maybe they think that they’re just like far more likely to be recourt. And some fraction of them are reconsidering their life of crime.

Jennifer Doleac: Yeah. I think there’s kind of two ways I think about this. So yeah, there’s this whole issue of how accurate are their perceptions of the likelihood of getting caught. And surely, to some extent, they’re overestimating because yeah…

Robert Wiblin: Most criminals aren’t caught most of the time.

Jennifer Doleac: Yeah. Yeah. But so to some extent that could be due to, you know, the way that people learn about this sort of stuff isn’t from reading academic papers, it’s from watching things like crime shows on TV and there like everything works immediately. And so it’s like magic. And so people might think “Ah, I’m in the database. It’s like magic. They’ll catch me”. So it’s possible they’re overestimating. And then you just need to count on sort of the technology progressing faster than people learn that it’s actually not that good. The other possibility is there’s this kind of prospect theory that people are just really bad at interpreting probabilities. And so if it’s like above 0.8, we’ll just call it one. And if it’s below, you know, 0.1, we’ll call it zero. And so something where it increases the probability sort of enough, it might just be something where people are like, the way that in practice humans interpret that higher probability just suddenly like pushes it to like almost a hundred percent in their minds and the way they respond to it. So I think that’s also a possibility here. There’s some sort of threshold or tipping point in terms of it’s a high enough probability of getting caught that I feel like it’s like basically a certainty. and so I want to change my behavior.

How ambient light influences criminal behaviour [01:46:56]

Robert Wiblin: Yeah. So on this topic of ways of preventing crime before it happens, there’s another paper of yours “Under the Cover of Darkness: How Ambient Light Influences Criminal Behavior”, and also there was an episode of Probable Causation which I found very memorable, which looked at putting very bright lights in crime hotspots in New York I think near public housing. They put like flood lights. So it was fairly extreme, but it was really dramatically increasing the lighting in these areas where a lot of crimes were being committed which found, at a minimum, a 36% reduction in nighttime outdoor crimes. And I think largely it wasn’t just a matter of substitution. It was really reducing crime. So what have you learned from your own research on lighting? What implications does that have? Should we be rolling out the LEDs in addition to our now better trained police officers?

Jennifer Doleac: Yeah. So my own paper with Nick Sanders is using daylight saving time as a natural experiment to measure the effect of ambient lighting. And so basically the idea is that on one day, the sun sets at 5:00PM and the next day it sets at 6:00PM. And so suddenly that 5:00PM – 6:00PM hour is light instead of dark. And because our schedules are somewhat fixed based on what time work ends and what time we’re commuting home and all of that, we can’t just adjust easily to that change in daylight. And so if we think that potential criminals respond to the likelihood of getting caught, then this should matter. So we have this conventional wisdom that you don’t commit crime in broad daylight. Right. And I think the idea there is that if you were to mug someone, say, when the sun is shining, they might see you coming. Witnesses might see you and be able to identify you later. Like, in general, you’re gonna be more likely to get caught for that than if you do it in the dark. And so we’re able to test that and find that when that evening hour is dark instead of light, robberies go up. So it does seem like light matters. Of course we don’t have full control over sunlight. We can move it around in that way, but we can’t just make more of it. So the policy–

Robert Wiblin: I guess you can move to a different latitude, although I suppose then you have to do it in winter.

Jennifer Doleac: Yeah exactly, somehow there’s some science fiction story that might come out of this, but yeah. So the paper that you’re referring to about the floodlights, or the streetlights, Aaron Chalfin was the guest on my podcast who was talking about this and one of the authors in that study. So they actually were able to randomize within public housing where these additional streetlights were placed. That sort of like the next best thing to adding sunlight is the man-made light. Yeah. And they found big crime reduction benefits. So I think again, the idea is you’re less likely to commit crime when it feels like you are more likely to be observed and identified later and might be punished for it.

Robert Wiblin: What’s the substitution towards people just committing crime later in the night when you’re under daylight saving time? Or was it just like an actual reduction in total crime?

Jennifer Doleac: No, it was a net reduction in crime. Basically what we would expect, you know, the standard rational theory of criminal offending perhaps might make you think, “Well, they’ll just commit crime in the morning instead because that’s where the daylight’s being shifted from”.

Robert Wiblin: They’ll just get up at 6:00 AM in the morning like early birds.

Jennifer Doleac: Exactly, get them on their to commute to work instead of the commute home… Yeah, I think it appears that criminals are not early risers, and so that does not happen.

Robert Wiblin: Gotcha. Yeah. Okay. So I’ve got a bunch of other ideas, or like other policy areas where we might better reduce crime before it happens: cognitive behavioral therapy — I think there’s been some positive results on that. I think preventing pollution. I think we have evidence that lots of forms of pollution that, I guess, were very prevalent in the United States in the past, and are still surprisingly common today have cognitive effects that make people more likely to commit crimes and more impulsive; I guess that’s known with lead, but it seems like it’s not the only case. And then possibly we could also talk about, you know, economic opportunity. How do government transfers affect crime. Are people committing crime sometimes out of economic desperation? I would be excited to talk about those if you have interesting things to say about them. But there’s all these other ways of reducing crime outside the criminal justice system that don’t impose the massive costs of sending people to prison and ruining their lives and harming the kids and their family and so on. And just eyeballing it, it looks to me like maybe we’re underinvesting in police inasmuch as they reduce crime. But it seems like we must be maybe underinvesting in these even more street lighting perhaps could be a really big deal. Is that qualitative impression right?

Jennifer Doleac: That is also my impression, yes. I think based on the evidence, we should be pouring money into education and summer jobs for teens and lead abatement. I think if anyone came to me and said like what’s the number one thing if I’ve got a million dollars to spend to reduce crime, what would you do with it? I’d say invest in lead abatement. Yeah. Pollution: I just interviewed someone for Probable Causation about the effects of air pollution on crime. Big effects there. I think there are bigger effects of air pollution on mortality. So a reduction in violent crime is actually like a rounding error in the social benefits of that.

Robert Wiblin: But even then it would be justified just on that alone.

Jennifer Doleac: There are tons of things we should be investing more in that would be cost-effective to invest in. And I think especially in the crime context, I work a lot on prisoner reentry and trying to understand how to reduce recidivism, and it turns out it’s really hard. It is much easier to prevent crime upfront. To prevent someone from committing their first crime, than getting someone onto a better track later once they’re sort of already in the criminal justice system. So the more that we can invest upfront in preventing anyone from their first criminal justice interaction, the better off we are as a society for sure.

Cognitive behavioral therapy [01:52:37]

Robert Wiblin: Okay. Yeah. Let’s talk about cognitive behavioral therapy and pollution and any kind of economic opportunity things. Are there any papers there that you’d like to highlight which maybe indicate how large the effect sizes might be?

Jennifer Doleac: Sure. I’m not sure I’m going to have the effect sizes off the top of my head, but the main–

Robert Wiblin: Just say “big”. It was really “big”…

Jennifer Doleac: Really big. Yeah. So the main paper on cognitive behavioral therapy is by Sara Heller and colleagues. There are other papers too. It’s not just kind of a one-off. CBT is a place where I feel like we have a fair amount of evidence. So cognitive behavioral therapy for folks who are listening is basically a form of therapy where you’re taught to kind of slow down your decision-making and think through the assumptions you’re making. Perhaps about the context or the script you have in your head about the way that the social interaction is going to go and questioning those assumptions and then perhaps make better decisions as a result.

Jennifer Doleac: And so, yes, there was a randomized controlled trial done in schools in Chicago, as well as juvenile detention facilities in Chicago, where the individuals that were assigned to get CBT were then followed over time and compared with people who were not assigned to get CBT. Again, it was randomized. And they found really big benefits. So the juvenile detention sample, they found really big reductions in the likelihood that those kids were reincarcerated in the future and really persistent effects. So, I mean, it’s just incredibly cost-effective. Like this program is not that expensive and the benefits are just massive. So that’s a place where, I mean, I kind of joke along with the lead abatement, which is not a joke. I tend to joke that like the other kind of hypothesis I have about how to reduce crime, it would be like just give everyone a therapist.

Jennifer Doleac: I feel like we’d all be better off if we all have a therapist. And I feel like cognitive behavioral therapies… Like I bet that other forms of mental health care are also effective. CBT isn’t quite mental health care, but it’s like in the vicinity. But I think any sort of therapy that’s helping people figure out why they make those decisions.

Robert Wiblin: Talk through their problems.

Jennifer Doleac: Yeah. Talk through their problems and figure out why they make the decisions they do and maybe make better ones that would probably benefit everyone. But especially people who’ve grown up, you know, with the trauma of violence in their lives and so on. So yeah. So CBT is a good one.

Lead reduction [01:54:52]

Jennifer Doleac: Let’s see. Pollution’s the other one. So lead. Yeah. There’s been a good amount of work on lead. So Kevin Drum has been a leading proponent of the lead-crime hypothesis that we have this mysterious drop in crime rates beginning in the early to mid nineties that have basically continued unabated through today.

Jennifer Doleac: Like suddenly, you know, crime was really high in the mid nineties and then suddenly like fell farther and farther. And it’s huge. Yeah. It’s just like crime is no longer a problem. Like the US used to be super dangerous and now, despite everything we were saying earlier about how high violent crime rates are, actually the United States is incredibly safe relative to the past. So Kevin Drum and others have sort of noted like if you kind of go back from the peak of crime in the mid nineties, 15 to 20 years earlier, that’s basically when gasoline was deleaded. So we had lead gasoline and cars to drive around and the exhaust from the cars would spread lead around the environment. And then at a certain point in time that was banned. And so now there’s no more lead coming out of the car exhaust.

Jennifer Doleac: And so, fast forward to like the kids that grow up now without lead in the environment and they no longer are committing crime, perhaps, let’s just say that as a hypothesis out there, the big drop in crime in the nineties is surely the product of a whole lot of things. I don’t think we have any conclusive evidence that that was the story. But it led to a lot of interest in the link between lead and crime. And so there is now other evidence that I find extremely convincing using a variety of approaches, all measuring the causal effect of both exposure to lead and the CDC recommended guidelines for anyone who is exposed to lead. Like the intervention that they do to try to mitigate that effect on things like educational outcomes, as well as criminal activity. And basically lead is just really, really bad and lead seems to cause high crime rates.

Jennifer Doleac: One of my favorite papers that isn’t necessarily… There are others that are more recent and kind of based on samples that are perhaps more relevant for policymakers today. But one of my favorite papers is an economic history paper where they compare places that had lead pipes or non-lead pipes. And then they also take advantage of the fact that apparently only like the water has to be a certain acidity level or like in a certain range for the lead to seep into the water. And so it was like a really nice, not just difference in difference, comparing places with lead pipes or not, but a triple difference where they layered on like you have to have a lead pipe and this certain acidity level of water. Those are the only places that’re treated really that like lead would get into the water supply.

Jennifer Doleac: And so you’re basically using other places with lead pipes as a control group and other places with similarly acidic water as a control group: things that might be different with other cities. And they find that the places with the lead pipes and acidic water had much higher homicide rates, dramatically higher homicide rates than other cities. I mean, I love just the mix of science and the history there that that is able to kind of provide a really nice natural experiment. And I also just think it’s a really compelling result.

Robert Wiblin: I guess we’ve already done a fair bit to reduce lead and I’m wondering presumably there’s still some places that that needs to be removed. There’s still too much of it. But I guess what, on the margin, do we need to do today in terms of pollution reduction?

Jennifer Doleac: Oh, so you’re right that like overall lead levels are much lower on average, but we do, I mean, in the US we have cities like Flint, Michigan that has major problems with lead in their water. Like this is an ongoing disaster in certain cities. So there still is a lot of work to be done there.

Robert Wiblin: Okay. Yeah. So you think lead reduction is the best buy maybe in crime prevention?

Jennifer Doleac: Seems to me. I mean, I feel like there is really compelling evidence right now that it not only increases crime, but it also dramatically reduces educational outcomes which can have long-term effects. So in general, it just affects your brain in such a negative way that it would be extremely cost-effective to invest more there.

Robert Wiblin: Yeah. Do you want to say anything about economics, or I guess economic opportunity and inequality and government transfers and things like that and the effect that that has on crime. Is that an important factor?

Jennifer Doleac: Sure. Yeah. So I think the best evidence on this comes… Well, so we can kind of go back to the divide between preventing crime and reducing recidivism, right? So on the prevention side, there’s some really nice evidence that things like head start programs and government transfer programs that increase wellbeing of children when they’re very young. So increased educational opportunities and health interventions, food stamps in the US providing better nutrition, all of those then lead to lower crime rates later. There also are a lot of policies and there have been policy changes in the US related to who’s eligible to receive those types of government transfers as adults and often a criminal record makes you ineligible. And so some people have taken advantage of those types of policy changes to see what the impacts of banning people with criminal records from receiving that kind of government assistance has on the recidivism rates. And it tends to increase recidivism. So, in general, in line with the idea that increasing financial resources tends to reduce criminal activity.

Diversity in economics [02:00:12]

Robert Wiblin: Interesting. Let’s move on and talk about the economics profession for a little bit. I follow you on Twitter, as our listeners might be able to tell, and one thing you did on there last year is post a lot of job market papers written by female economics PhD graduates specifically, to try to get them, I guess, noticed more and raise their chances of getting a good academic position. And recently you’ve tweeted about a bunch of ways you think that the profession is not as welcoming as it could be to people of color. Yeah. How do you think having more and more women and people of color in economics would improve the profession? I guess one obvious part that occurs to me is of course people have focused on different questions. Questions that particularly concern those groups that otherwise might’ve been neglected. I suppose, you know, economists think more about crime than they used to and police, but maybe if there were more people of color in the profession, we would have got onto this research agenda decades earlier. Yeah. Do you have any thoughts on that?

Jennifer Doleac: Yeah, so I think you’re totally right that different people with different life experiences will not only ask different questions, but have different hypotheses about what the answers might be. And especially as we’re trying to figure out which things should we be testing that could improve policing and so on. Having people who, again, have more or different experiences with police officers than the people who are currently in the economics profession might lead to testing of different hypothesis. The other way is just sort of a human capital story. If we want the best and brightest people in the profession, then we can’t just exclude large groups, right? It’s going to be socially inefficient for us whether there’s intentional policies, or just tolerating bad behavior, or whatever else, if that discourages the best and brightest from joining the economics profession. That is a detriment to the profession. That makes us all worse off.

Jennifer Doleac: It means that amazing econ papers that would have been written might not get written. And so I want colleagues who are the best and brightest people and knowing that there are people. Like I get emails all the time from college students or graduate students who are watching what’s happening in the economics profession and wondering if this all means that they would not be welcome there and maybe they should go apply their amazing skills somewhere else. And that would be a huge gain for whatever that somewhere else is and a huge loss to us. And so my response is generally the fact that we’re having these conversations out loud right now in economics is actually a really good sign. It does not mean that economics is worse than most of the other fields that they might be considering like tech or business. All of these fields have their own problems.

Jennifer Doleac: Yeah, but I think the fact that we’re having these very difficult conversations very publicly, and that even junior faculty are willing to stick their necks out and register concern about senior people in some cases, I think is ultimately a very good sign that they’re more confident than they would have been in the past that this is not going to hurt their careers. And so the first step is reporting and bringing the problems to light. And I think we’re doing a lot of that right now. And there’s a lot of energy. Back when we were allowed to travel, whenever I would go around and give talks, this was a conversation almost everywhere I went, whether it was over lunch or dinner, and office meetings, you were like, “Wait, what is your department? What have you heard about? What are people trying to try to get more women and minorities”? Either it’s grad students or faculty, or make people feel more included.

Jennifer Doleac: And everyone’s just sort of been brainstorming. Not everyone, but a lot of people are in brainstorming mode right now. And I feel like this is, in so many ways, if anyone can solve a problem like this, it’s going to be economists because we not only think about how people respond to incentives and why people make the choices that they do. But our toolkit allows us to test whether things work. And so we can generate hypothesis, but then we can also go test them. And so… I don’t know. I’m confident. I’m an optimist on this. I feel like there are a lot of problems and I have tenure, so I have the job security that is necessary to be comfortable pointing them out. Yep. So I might as well use it for good, but I’m also just very optimistic that we’re moving in the right direction.

Robert Wiblin: Yeah. I actually almost can’t think of another area where it seems more important to have people with a wide range of life experiences represented in order to get to the truth than social science research. It’s going to be devastating to not to.

Jennifer Doleac: Totally. Yeah. And especially, I mean, in economics a lot of people really don’t like economists because people listen to us. Right. And we have a seat at the table at a lot of really important tables. And that’s partly because we do have this tool kit that allows us to test what the effects of policies are and quantify them and help policymakers think about trade-offs and all those things. And I might be a little biased, but I think the economics toolkit is really important and really helpful.

Robert Wiblin: I guess it’s a bit of a litmus test for economics if it can’t cure itself of these problems then it’s like, what use is all of this social science and causal identification if you can’t figure out the right treatment.

Jennifer Doleac: Yeah, by revealed preference I think all that stuff is important. But yeah, because we have these seats at these important tables, we need to make sure that we have good answers and that just for like moral and ethical reasons, we should make sure that people have a fair shot at being at those tables. And even if you don’t care for some reason about… I guess I lead with the efficiency arguments because lots of economists don’t really care about the ethical or moral arguments. They want to know about the efficiency. And I think there’s a very strong efficiency argument to be made for increasing diversity. The status quo is just so inefficient, but I think most people are also persuaded by the moral argument.

Robert Wiblin: Yeah. I’m always reluctant to report information or to highlight information that might discourage people from going into jobs where they’re currently underrepresented. Because then you’re just potentially reinforcing the problem. But it’s also important that people have an accurate picture of what their professional life is going to be like and more barriers that they might face. So given that, what would you say it’s like being a woman in the economics profession with things considered. And yeah, what are some of the specific things you’d like to see changed to make it more and more appealing or accessible, even if it’s not worse than other similar professions?

Jennifer Doleac: Yeah. Good question. I mean, I think there’s been a lot of research that has come out in recent years about how women’s research tends to just be given less of a benefit of the doubt than the research of white men. And women tend to get less credit for the research they do do. So if you have a male coauthor, he will get more credit and he’s more likely to be promoted to tenure than you are which suggests that the promotion committees give him more credit for the work. Papers that look equivalent on a variety of dimensions are, if you compare two very similar papers that are published in our top journals, one by men and one by women, the women’s papers are actually cited much more often, which suggests that there’s a higher bar, to the extent that citations are a proxy for contribution, that suggests that that marginal paper is actually much better than the marginal male paper.

Jennifer Doleac: And so that suggests that the bar is higher basically to get in as female authors. There’s a bunch of evidence around that. So those are the facts that I would like to see changed. I think the big question, of course, as in every tricky policy area is what to do. Like how do we change it? And I think that is what people are really grappling with. So for the past year and a half, I’ve been collecting data on who gets invited to give seminar talks in econ departments. And part of that is because I actually think that those kinds of opportunities are really important and are a lever that we have the ability to push on. People are like, “We should hire more women and we should hire more minorities”, but if you have a limited pool and you don’t get to hire that often, it’s hard. That’s just really hard.

Jennifer Doleac: But if you want to invite more female speakers or invite more minority speakers, that’s easy. Everyone has seminar series. We can do that every year. And I think it’s important. Seminars are how researchers disseminate our research. It’s how we get feedback. But it’s also how we build networks. And I think the networking aspect of it is incredibly important. So when people ask me what to do, I’m like, “Diversify your seminar series”. Like it just feels like low hanging fruit to me. But people are experimenting with all kinds of things like role model interventions. An example of something that doesn’t work is gender neutral tenure clock pauses. So in general, for academics, if you have a baby, your tenure clock is paused for a year. They moved from just having that for women to being for men. It turns out when men get the tenure clock stop or pause also, they just use that extra gear to do more work. So that policy actually just increased tenure rates for men and didn’t do anything for women. So it actually made the problem worse. So anyway, so it’s just another example of the potential of unintended consequences and needing testing and being willing to be wrong about what it’s going to take.

Robert Wiblin: Yeah. Obviously you can’t speak to being a woman of color in economics yourself, but are there any other forms that you think might attract a wider range of people from all different races to consider a career in economics?

Jennifer Doleac: Well, I mean, I have to say that it was part of my motivation for starting my own podcast was just to kind of showcase the cool work that economists do and the kinds of things that we think about all day that isn’t, you know, GDP and money. I mean, I think even when I was in college, I always loved economics and I was an econ major. I just sort of assumed I would be an investment banker because that’s what you do with an econ major. And if you don’t want to do that, then it’s not really clear what the options are. And now that I study crime for a living, it’s so obvious to me that the econ toolkit is applicable to so many things.

Jennifer Doleac: Like if you care about policy in any way, it is extremely valuable to you in whatever type of policy you care about to have this toolkit at your fingertips. So making that case, I think there’s a lot of energy that goes into just showcasing the variety of policy problems that economists study and contribute to solving. And I hope that that helps make us more appealing to a broader audience. But I think when people who are currently in the field can say good things about it, I feel like they’re treated fairly, that would probably also help the pipeline. And so moving in that direction seems good too.

Quality social science research [02:10:44]

Robert Wiblin: Let’s talk about another aspect of the economics profession. I guess it seems to me like there’s been a big increase in higher quality social science research recently. I’m not sure whether it’s just an illusion because I’ve started listening to your show, but I guess this all looks to me kind of like a post-credibility revolution wave where suddenly… Well there’s an explosion in the use of these natural experiments to try to actually get answers to causal questions in economics and unrelated areas. Do you think that sounds right? And I’m interested to know how much do you think we can trust this literature we’ve seen? I suppose, lots of other research in psychology and other aspects of social science and I guess all of the results maybe from observational studies now seem more suspect than we used to think they are. How much should we believe all of this research?

Jennifer Doleac: Well, I think responsible researchers are always careful about thinking about generalized ability and you don’t make policy based on single papers. That said I do think that economists approach to the concerns about like p-hacking and so on is better, I think, than a lot of other disciplines. So basically what economists do, and people gripe about this all the time as we expect… You’ve got your 30 page paper, and then we also expect like a hundred page appendix with robustness checks where you show me what happens if you did every other thing for every judgment call you made. So if you had some cutoff at 25%, I want to see what would have happened if you’d made it 20 or 30 or 35 or 10, so you just have this appendix full of all these other graphs that show you that your estimates aren’t actually based on you cherry-picking the one result. And I feel like you just cannot publish a paper in economics anymore if you can’t demonstrate that to reviewers. And I read a lot of other papers from other social science disciplines and it’s like, they don’t even have a graph. Like the main result has much less of those kinds of robustness checks. And I read those and I’m like, “I have no idea what to do with this estimate. Who the heck knows”?

Jennifer Doleac: And in psychology there’s been this big emphasis on preregistration and pre-analysis plans. And that has always struck me as not… Like I have friends who work in psychology and they’re like basically what winds up happening is it doesn’t lead to the theory driven hypothesis that everyone hopes it will. People at the higher resource schools just go run a whole bunch of pilots, find the thing that works and then rerun that and preregister that and rerun that one. I mean, if you want replication, I guess that works. But it’s not exactly the theory driven science we want. And the pre-analysis plans are just always… With these observational studies in particular, there’s so much learning that goes on along the way. And I totally understand the concerns about p-hacking and, you know, trying to split the sample enough times that you find that the one group that you’ve left stars for. And I think again, good researchers and good reviewers are always on the lookout for that. And I think it’s tough.

Jennifer Doleac: It’s very difficult to publish papers if it seems like that’s what you’ve been doing. But it’s always struck me as bizarre that knowing how much learning goes into these types of papers, where like you run your first regression and you’re like, “Oh, that’s really weird. I wonder why that is positive. I would’ve expected…” And then it’s like, “Oh, that variable, they changed the way they coded it in the middle of the sample”. Or, you know, just all the stuff that comes up along the way like that. And so the idea that the first regression I would’ve thought to run is somehow the most clean or the best one just seems crazy to me. And so I talk to people about pre-analysis plan and they’re like, “Oh, but you can update it”. But referees don’t update.

Jennifer Doleac: You get accused of like… If you’ve changed your pre-analysis plan, that seems wrong. So anyway, I really like that our approach is this. I just want to see it every other way for every judgment call you made. And it’s very difficult. It doesn’t make it impossible to come up with a result that’s not going to hold, but I think it minimizes it more than any other approach I’ve seen. And then ultimately, if the theory is correct, then we should see similar results hold in other contexts. And yeah. And that’s where more work is always helpful.

Robert Wiblin: There’s a bunch of different kinds of natural experiments. There’s policy lotteries. There’s random judge assignment is used a lot here. I guess you’ve got staggered rollouts is another classic one. Or like the place that just got funding versus the place that just got rejected for funding. That kind of thing. Are there any methods that you’ve learned over time to give more weight to and to believe more and other ones that you’ve learned maybe to be a little bit more skeptical of?

Jennifer Doleac: They all have their pros and cons. So I mean the randomized controlled trial people often refer to it as the gold standard, but it is not feasible in a huge number of contexts, especially public safety contexts. And so it’s amazing to me how often I say we need evaluation in some area and people are like, “Oh, we can’t do an RCT”. And it’s like, we don’t need an RCT. There are other ways to try to measure the effectiveness of this program than randomizing. And I think that’s not a message that’s gotten out to most people. So let’s see. So yeah, I mean, regression discontinuity. So that’s like you just barely got in versus you just barely didn’t, based on a test score or something. Those provide really, I think, high quality causal estimates, but they’re only applicable to the people who’re right around the threshold.

Jennifer Doleac: So if you have a threshold that is a really weird population for some reason, then that’s actually not all that useful. And so the best studies also make the case that that marginal population is important for some reason. Like it matters what happens to those individuals. Let’s see. The staggered rollouts can be good. I think a kind of similar sort of design, or like difference in difference, or panel difference in difference designs where you have policies that are adopted in different times in different places. So in some ways, those kinds of studies are the most feasible in a lot of policy contexts, especially in the criminal justice context where you have a lot of variation place to place in the type of policy that’s adopted. And so it’s a good tool to have in the toolkit.

Jennifer Doleac: That said, there’s been a tremendous amount of econometric and methodological work in the last couple of years. I’m trying my darndest to keep up with about these panel difference in difference designs and the potential pitfalls and what the coefficient actually means and how it’s kind of a weighted average of the different effects over different years and all this stuff that it’s become much more complicated and we’re going to start requiring a whole bunch more robustness checks in that appendix very soon. So, yeah, I think they all have their pros and cons, but I think in general, I think of them all as being really good potential tools in the toolkit, especially when you’re talking to policymakers about all the possible ways that you might be able to evaluate their program. Sometimes a randomized control trial works out, but it’s good to have, you know, other options in your arsenal so that you can continue the conversation and get at least a pretty good estimate, even if it’s not the perfect estimate.

Robert Wiblin: Are there any tell-tale signs of which papers you should give more weight to and which to less than maybe people don’t appreciate, or that kind of you’ve learned to use that you think you might want to pass on to us as readers of literature?

Jennifer Doleac: So in general I think that if someone can’t explain clearly and intuitively to you as a smart lay person what the natural experiment is, or what the intuition of why the control group or comparison group is a good counterfactual for the treatment group: if they can’t explain that clearly to you, then they don’t fully understand it themselves. And I would be very wary of that. There are a lot of papers out there that just do lots of really complex math and lots of really complex computations and there’s some crazy model or something and then on the other side comes a number and you’re like–

Robert Wiblin: It’s a sausage factory.

Jennifer Doleac: Exactly. It needs to be extremely transparent. And so, yeah, I think in every good experiment paper like this or paper that’s trying to measure a causal effect, there is a thought experiment in mind where there’s some sort of ideal experiment where you’re randomizing in the lab that you’re trying to approximate in the real world. And so if you cannot talk through very clearly what your treatment group is and what your comparison group is relative to that ideal experiment, then I would be very wary of that paper. And similarly, I think most economists at this point, like we just use OLS in our regressions. Like the fancy models and all the hierarchical models that other fields are using. I feel like, again, it adds a bit of sausage making process where you really–

Robert Wiblin: Draw a line! Just draw a line.

Jennifer Doleac: If you’ve got a good experiment, put that stuff in the robustness check in the appendix, right? To show that with all the fancy modeling, you basically get the same answer. But you should get the same thing from OLS; if you get different answers, then I think that’s a problem.

Robert Wiblin: Yeah. I couldn’t agree more if someone says cutting edge statistical method, I’d say like reach for your wallet.

Jennifer Doleac: Yeah.

Robert Wiblin: So cutting edge is not great with these things or the more complicated, the more you worry about chicanery or people just not knowing what they’re doing.

Jennifer Doleac: Or people not knowing what they’re doing. Yeah, exactly. I think the other big one that’s sort of along the same lines is, again, like the standard economics at this point is you should be to show in a graph what your effect is. And there’s no way you can visualize an effect at some threshold or something, or some change at a certain point in time. And like you see the coefficients drop at that. If there’s not a graph like that in the paper, you should be more skeptical of the result because you just don’t know. You want to see that the change happened when you think it’s going to happen. And not that maybe it happened two years after the policy change but like, it all goes into the average. So we’ve become big on graphs and transparency and just keeping it really simple in economics and I think that’s a good thing.

Robert Wiblin: All right, we’ve got 2 minutes left. I guess a lot of people feel incredibly depressed or pessimistic about things at the moment. Hard to look at the news and feel positive about it. I guess we usually try to end on a positive note on the show, but I guess I’ll ask actually your honest opinion. Are you optimistic about whether the US criminal justice system will be more fair or kind of less cruel and more effective in 10 years time? Is it possible to put a positive spin on this?

Jennifer Doleac: Absolutely. I’m very optimistic on this. I think not only is it a golden age for research in this area, I mean, I think one of my favorite things about working in this space is that I routinely work with people who are both far more conservative and far more liberal than me. We’re all at the same table, trying to reach the same goal. This is like the one policy space I know of where there isn’t a clear left and right on different policies. Everyone agrees the current system, almost everyone agrees the current system is far from what we want it to be. And I think that because there is no defined left and right in terms of the changes we want to make, that means everyone’s more open to just knowing what the evidence says. So this is a space where I feel very valued as a researcher, which, you know, as a researcher, I appreciate. But I think should give other people confidence to that evidence is going to inform the policy changes.

Jennifer Doleac: Yeah, I think there has been tremendous momentum for criminal justice reform broadly in the United States. And that has been moving. This is all state and local. So we don’t depend on the federal government making the changes. And the recent protests and conversations we’ve been having about policing are just, I think, the next step in that. I think there’s going to be real change that comes out of this. And I don’t know what it’s going to be yet. And that’s exciting, but I’m very optimistic that 10 years from now we will be living with a much better criminal justice system.

Robert Wiblin: Well, I really hope you’re right. Thanks for everything you’re doing to try to bring that about. My guest today has been Jennifer Doleac. Thanks for coming on the 80,000 Hours podcast, Jennifer.

Jennifer Doleac: Thank you so much for having me. This was great.

Rob’s outro [02:22:30]

Robert Wiblin: If you’d like to hear more from Jennifer you can of course subscribe to her show, Probable Causation.

Her latest two episodes are on the impact police have when on the street and in schools.

Other episodes have covered racial bias in police investigations, street lighting, prison bail reform, and youth employment programs as an approach to violence-reduction.

The 80,000 Hours Podcast is produced by Keiran Harris.

Audio mastering by Ben Cordell.

Full transcripts are available on our site and made by Zakee Ulhaq.

Thanks for joining, talk to you again soon.

Learn more

Policy-oriented government jobs

Related episodes

About the show

The 80,000 Hours Podcast features unusually in-depth conversations about the world's most pressing problems and how you can use your career to solve them. We invite guests pursuing a wide range of career paths — from academics and activists to entrepreneurs and policymakers — to analyse the case for and against working on different issues and which approaches are best for solving them.

The 80,000 Hours Podcast is produced and edited by Keiran Harris. Get in touch with feedback or guest suggestions by emailing [email protected].

What should I listen to first?

We've carefully selected 10 episodes we think it could make sense to listen to first, on a separate podcast feed:

Check out 'Effective Altruism: An Introduction'

Subscribe here, or anywhere you get podcasts:

If you're new, see the podcast homepage for ideas on where to start, or browse our full episode archive.