In this BSC podcast, Salimah Samji, Director of Building State Capability has a conversation with Professor Matt Andrews, Faculty Director of Building State Capability to discuss public policy implementation failures.
To learn more about the Implementing Public Policy executive education course, visit www.hks.harvard.edu/EE/IPP. To read the Public Policy Failure paper, visit https://bsc.hks.harvard.edu/publications/public-policy-failure-how-often-and-what-is-failure-anyway/.
// https://bsc.hks.harvard.edu ///
Interview recorded on January 18, 2019.
About Matt Andrews: Matt Andrews is Senior Lecturer in Public Policy. His research focuses on public sector reform, particularly budgeting and financial management reform, and participatory governance in developing and transitional governments. Recent articles focus on forging a theoretical understanding of the nontechnical factors influencing success in reform processes. Specific emphasis lies on the informal institutional context of reform, as well as leadership structures within government-wide networks. This research developed out of his work in the provincial government of Kwa-Zulu Natal in South Africa and more recently from his tenure as a Public Sector Specialist working in the Europe and Central Asia Region of the World Bank. He brings this experience to courses on public management and development. He holds a BCom (Hons) degree from the University of Natal, Durban (South Africa), an MSc from the University of London, and a PhD in Public Administration from the Maxwell School, Syracuse University.
Episode 4: Public Policy Failure
[00:00:04] Hello and welcome to building state capability at Harvard University's podcast series. In this BSC podcast, Salimah Samji, Director of Building State Capability, has a conversation with Professor Matt Andrews, Faculty Director of Building State Capability to discuss public policy implementation failures.
Salimah: [00:00:28] Welcome, Matt. It's always a pleasure to be able to do a podcast together with you. Today I wanted to talk about implementation failure. You know, we hear that there are so many amazing policies that are out there, but there is a large percentage of them that just fail. Why? Why do we have so much failure in public policy implementation?
Matt: [00:00:49] You know, it's kind of an interesting question, because on the one hand, if you come to a school like our school, people are going to talk a lot about the successes. And we teach case studies about everything that goes right. If you're out on the street, people often talking about everything that's going wrong. So, you know, it's almost these two narratives that you hear. On the one hand here, we celebrate government and we celebrate public policy and out there people complain about it a lot. And you just have to read the newspapers and you can see that second narrative, that second narrative about things don't work very well. And I think when you put the two together, you realize we have a lot of very good ideas that we want to be successful policies, but somehow they don't get there often. And the question is how often? Right. And we've been thinking about this very seriously over the last couple of years, as you know, partly inspired by people like Frank Fukuyama and Stanford and others who've been identifying that, you know, we don't necessarily teach well about taking the ideas and executing the ideas. And I think that this is an important gap thinking about failure. So, you know, the first thing that I've been asking myself as well, how often do we not succeed? And that's an interesting question. And we start to ask the question. You say, well, how often do we succeed? Which side is right here, you know? And the first thing that to observer, it's actually very hard to see that because there are very few governments that actually evaluate do we do well? Do we succeed? Very few agencies or entities across the world that actually look at this question. And what do you think that is?
Salimah: [00:02:22] Why do you think that is?
Matt: [00:02:22] Well, you know, I think that, for one thing, it's very hard to do. And the first thing to think about public policy is if you want to do public policy, it's the hardest thing in the world to do now. I have a lot of friends who are entrepreneurs and they work in the private sector and they will be shocked that I've said that. You know, I came to this stuff through economics and the economics you learn. Why do you need government? Well, you need government because of market failure. Right. So the very idea that you need government is, well, there's some things that the private sector can't do. There's some things that the private sector fails at doing it under provides. Right now you're saying, well, OK, the government is the right place to do it. But what you're doing is you're saying let's take the things that the market can't do and it's given to the government and then we're surprised that the government can't do it either. So I think the first thing is these things that we could hide problems. Oftentimes they involve doing things that we don't really know how to do. Bringing people together in ways that we don't know how to bring them together and engaging in political spaces that are fraught with complexity, that are often very adversarial and developing mechanisms that allow us to do things that we don't know how to do and that may be impossible. So in one sense, I think that's one of the reasons why we don't always assess success because we're not really sure how to do what we're doing. And worse than that, we're not really sure how to evaluate if we're being successful in what we're doing either. What does success look like?
Salimah: [00:03:44] So you have this paper that you've written on an implementation failure and you've kind of looked at the World Bank and some of its projects. Can you share with us some of the findings that you had from looking at at least an organization that's looking at failure and success and evaluation?
Matt: [00:04:04] So, you know, the World Bank is interesting. I used to work in the World Bank and we're interested in development, but it is one of those organizations to give a lot of credit that actually evaluates every project that they do. And, you know, people would say, well, this isn't a government. Yes, but it's a public policy organization that works with governments all over the world. And if you look at the composition of the World Bank, it kind of looks like a government. There's the people working on health. The people who work in education. There's almost every department you could imagine a government, the World Bank does that right. Now, they support these projects and they fund them like you would through a government budget. And then they evaluate them. And the first thing to say is, you know, some people read the article and think I'm being nasty to the World Bank. Unfortunately, I'm looking at them and I'm asking the question, how often do you fail? It's not being lost at all. I can't find any other organization that evaluates every project that it does and that actually puts that information out so we can look at that. So from a researcher's perspective, it's just a wonderful thing, it's a goldmine. The interesting thing is that when they do the evaluation, the evaluations departments actually asks a whole lot of interesting questions. And they have been filled with people who are very serious in thinking about this. What this failure look like. They ask questions about always satisfied with how the project is going?. Is it going on time? Is it on budget? Is it addressing the objectives that we had at the beginning? They also ask questions like is it meeting the broader development impact objective that we were thinking of, for instance, you might say we are doing a project to build schools and the immediate objective is to build 5000 schools in five years. And so they would ask that we satisfactorily do that within the budget that we have, etc., which are questions that you would call like project management questions or product questions that we produce the product. Right. But the reason why you're building the schools is so that people can learn. Right. And that's the development impact. Right. And every government has these kind of things where in one hand we're allocating money and we're doing in a period of time when we're producing a specific product and we want to know, do we do those things? But we're doing that for the reason. And that's a second question. And the World Bank asks those questions that at least the evaluators do. They also ask, is it sustainable? Great question. They ask a bunch of other things. And on the evaluation side, they really have thought about this stuff, you know, as sustainable. They ask questions like, you know, is this politically sustainable? Important question. When the World Bank as an organization, not the evaluators report on these findings, they only report on the satisfaction. They only reports on did we do it on budget, did we do it on time and did we reach the objectives. And on that, for every project they assess on different criteria and this one, they they report on the percentage of projects that are considered moderately satisfactory or better, and on that they 75 per are ticked off with successful. That's a large percentage. It's good. I mean, any organization that's good. And basically what they're saying is we committed to do this project. And what we can control is how the money was spent and effort was spent as we promised that it would be spent as in any good project management organization. Right. And they're saying 75 percent of the time we do that. The second question, you won't find the managing director reporting on every year of do we think that what we produced is going to improve the development impact?
Salimah: [00:07:32] Is it going to solve the problem?
Matt: [00:07:35] Now the evaluators are evaluating that. They are asking the question, well, turns out 75 percent moderately satisfactory, above only 49 percent of the time are they saying are we actually going to impact things. They are asking the question, but they are not speaking about it now. It's a really interesting thing because you could say, well, it's kind of obvious the 75 percent sounds better, the 49 per cent it doesn't sound that good. I think there's something else going on. Organizations have become project management organizations. And if you're a project manager, an organization, you're focused on the project and the product success. Did we do it on budget to do it on time and did we produce what we said we'd produce, because that's what we're accountable for and that's what we planned for and that's what we can control for. The thing of then saying, is it going to be sustainable? Is it going to impact things? There's a lot of uncertainty there. There's a lot of stuff that we as the organization in this case, the bank or the government, whichever entity doesn't control. We can't control those things because there's politics that comes in. Maybe this surprises. Maybe there's shocks. All right. And those are things that we say, well, we can't really be held accountable for those things in the scope of our project, because that's how fair to us. Now, the difficulty with public policy is that that's kind of where the rubber meets the road. Because it's not about building the schools. It's about having the kids learn. And the difficult thing that we have is when we evaluate like this, I think we end up with a lot of schools. But the big question mark about learning and it's not this is just one example, right? We could give a lot of examples. We could say, well, in Flint, Michigan, we end up with water treatment plants, but we still have dirty water. Yeah, we could talk about youth unemployment in certain parts of the United Kingdom and say we end up with a lot of programs and projects where we spend the money as we promised to. But the thing that we did didn't actually solve the problem. And because of the project that we had, we didn't have the scope to adapt and change and do something different.
[00:09:33] So it's almost kind of an interesting thing that the way that we are evaluating, trying to be accountable to the taxpayers, trying to be accountable to our bosses may be the thing that actually precludes us from implementing the thing that actually solves the problem. And that's kind of one of the messages of the paper and then the papers kind of saying we need to be evaluating these things, but we need to be incorporating this other question of do we actually think we're going to solve the problem more as well? And that's kind of a big question mark as to how you do that.
Salimah: [00:10:03] Yeah. So do you wanna share some thoughts you may have on how does one get out of this? Like what can be done differently? Because the examples that you give are exactly what we see. And we work in so many countries and we teach here at the Kennedy School and there are so many students here and said this is a recurring story that you hear. Probably every country, every city has examples where there are doing lots. So if you ask them, are you doing something about this problem? The answer is almost always yes. Is it working? Probably not. Or not as well as it could. How do you get out of this trap?
Matt: [00:10:35] It actually where we started off and said we live in this world where half of the people are saying we're doing well and then the citizens are saying you're not. I think this explains it because what we're doing is we're seeing but of course, we building the schools and they say, yes, but we're not learning. Right or. But what we have this kind of youth unemployment program and we've been moving the money and we've hired people and we're doing a good job and we're excited and people saying the youth are still unemployed. Right. Right. And so we're evaluating ourselves in a way that says legitimately, we're doing a good job, but we're not solving the problem because of the gap. Now, it is an interesting thing. How do you get out of this? Because this leads to lack of confidence in the state, even while the state thinks it's doing a good job.
Salimah: [00:11:15] It's like a spiral down. It's a self-fulfilling prophecy that just continues over and over again?
Matt: [00:11:23] You know, the easy way out of a spiral is to spiral the other way.
Salimah: [00:11:27] Oh, yeah. Oh, I like that!
Matt: [00:11:29] Kind of say, you know, if you want to change a negative feedback loop, throw some positive feedback into it. Just the thing that's really tricky. If we look at these projects and you say, well, just to bet on the project, we were already being 75 percent successful in this project in the way we're being successful. Now, when you look at these documents, these assessment documents and they say, well, how do you do better? They would say plan better. They'd say execute on the plan better, move the money better. Now, what they're doing is they telling you how to be more successful in the 75 per cent measure. And that's what they're doing right now. They're not telling you how to do better on the 49 percent. So it's saying, yes, we didn't build some of the schools. So let's make sure we finish all of those next time around. Let's do the things we can control. And usually that's where the advice on doing better comes from. And, you know, we teach that here. We say to the students here, be even smarter with your plan. That's what we say to them, control even better. What we need to be recognizing is that the issue with failure is on the things that we can't plan for.
Salimah: [00:12:34] Which is pretty much if you're working with complex problems, which are almost all of these things, you have no control.
Matt: [00:12:40] That's exactly what it is. So we're delivering on the complicated stuff that you can plan and control. And we're struggling on the stuff that requires you to be dealing with complexity where you're adapting and learning and actually the way in which we are advising people to do better on the project makes it harder for them to do it. So an example would be in many organizations if if I look at the World Bank assessments, one of the things that you'll see very commonly is they'll say if things didn't go well, it's often there was a political surprise something happened. And then they'll say, how do you deal with that next time around? Because the evaluators and again, the evaluators do a very good job is they'll say you need to do a better political economy analysis at the beginning. And so I'm like, okay, what you're telling us is that we need to have done a better evaluation of what was going to happen now four years ago. But we don't know what's going to happen now. And when you do that evaluation, you do the better political economy analysis that gives you more confidence in your plan. Which means that when the surprise happens, you're actually going to make a bigger mistake because you can't adapt. The issue is not through better plan to better control the issue as we need to build in more scope for adaptation. We need to have an approach where we recognize the things that are complex that are not easily planned for them. We need to have a better way of doing those things. So more flexibility, more flexibility. So our approach in that sense needs to say, OK, what is it that we don't know? What parts of the project do we need to learn about and how do we develop a project management methodology that allows us to learn as we go and to adapt? It doesn't have to be your whole project, but it needs to be part of your project. When you can't plan, don't plan, right? Right now, it doesn't mean, you know, you just do nothing. And this is where we come into our work on problem driven, iterative adaptation. We say develop a methodology that allows the facilitated emergence of the solution in a structured, responsible, accountable way. But that actually builds in the mechanism for learning and adapting.
Salimah: [00:14:38] Because the truth is, you don't know what's going to emerge. You don't know what the solution is. And you want to allow the space for this to just emerge as we've seen they do.
Matt: [00:14:47] And that's exactly what it is seeing. This gap, this gap between 49 and 75 percent is all about stuff that surprised us. Yes. And to say well planned, better for the stuff that surprises you, it's like, well, that's silly, you know, it. I mean, it's not totally silly because there are some cases where people had bad plans. But again, to give credit to the World Bank, most of the projects are well-planned. I mean, I mean, they have very good experts. They're planning things well, they're doing it with the government. This is not a city organization. And I would say this. Most governments I work with, they have good planners. They have people who do this thing where things fall apart is all these surprises. And one of the things that we teach here is, when you are thinking about your project, you need to ask how much uncertainty is there? And if you find that there's a lot of uncertainty, you're saying there is complexity here that we can't plan for. Because if it's uncertain, how do you plan for it? Right. And, you know, we'll say to people. How much is it that you know, that, you know, how much is it that you know that you don't know? And then how much is it that you don't know that you don't know?
Matt: [00:15:47] And the reality is there are many, many unknown unknowns. And so if you have unknown unknowns, you can't plan for those things. So even asking those questions, the beginning allows you to work out. Is my project one where I should really be more adaptive? As I move on, where I should really think carefully. The beautiful thing about management is that management gives us many ways to deal with many different problems. We have a plan and control methodology that we can use when we know a lot of things, right. And even if we know what we don't know, we can use plan and control because there's risk mitigation that we can bring into it. And that's what a good project does. The second thing we could do is we could use agile or adaptive methods where we know what we want to do, but there's unknowns about how to fit it to the context. Right. Right. There's things that we know that we want to use a new I.T. system in the health sector, but we're not exactly sure what the users in the health sector can or can't do. So we can use an agile method to kind of build that solution as we go. Right. Then there's other methods that are actually really good if we don't know anything and if we don't really understand the problem. Well, we don't understand what we want to do. We don't understand the context. Well, this is where facilitated emergence comes in, where we can start with the problem and we can build our way to the solution as we move along. All of these are management mechanisms. Problem that we see in many governments is that you have a lot of problems with lots of unknowns that require facilitated emergence or with significant unknowns that require agile and adaptive. But everyone is using planning control.
Salimah: [00:17:18] Why do you think that is? Because I'm sure some of them know. Right. And we run into people like this all the time, that they know that their plan and control method is probably not going to work as well. But they do it anyways. Why?
Matt: [00:17:29] I think a few things. I think one thing is that public organizations have to be accountable. And so we think in order to be accountable, we need to be telling people everything we're doing and we need to spend two years planning it out in advance and we're not going to get it done. And I think there's some truth to that. I think another thing is that, you know, we have false confidence in our abilities. I mean. I worked in government. I worked in the World Bank, and in all those times, sometimes I think I'd say my solution is so good that I'm pretty sure everyone's just going to get on board. Right. And it didn't happen that way. But I really kind of believed that. Well, meaning, you know, as as it may be, I think the kinds of people we train in the way we train. I don't think we train people to learn that there are other options out there. So people look at something and they say, really, there's a lot of uncertainty. I don't think the plan is really going to work, but they haven't been taught any other way. So if you go to the World Bank and you say to people, how do you do a project, they're going to all sit down and they're going to say you need to have a pre identification phase and identification phase, an appraisal phase. It's the same for everything. That's how the organization works. That's how people are trained. Many of the people who work in public policy, whether they're in the World Bank, whether they need government, it doesn't matter. They haven't necessarily been trained in alternative ways of implementation. They've been trained in doing policy analysis. Right. And then they think that's going to be the answer for everything. Even if they've seen that, it's not the case. I think people sometimes just don't know. It's one of the reasons why we're trying to say to people, the first thing you need to do is identify what your problem is. Often when we teach people about that, they say, well, there's a method for doing that?. Yes, there it is. And then we say to them, the beautiful thing about management is that there's different options to match the different problems. And they say, wow, is that really the case? It really is. So sometimes I think it's that simple. It's just giving people new options, saying to them this things that you can try. And we have found that that really helps people in addressing these things in better ways.
Salimah: [00:19:25] Right,you could even think about it as one problem, having different management methods you use for different parts. Right. It's not that one or the other. You can have a project with some plan and control, some facilitated emergence or some adaptive management combination of it's not thinking about, oh, my project has to fit into one of the buckets.
Matt: [00:19:44] Exactly. If you think about an education project, you could have an education project where you're building schools and then in each school you want to have some kind of IT based system where the teachers use in our teacher based system to engage with the students. And then you want to introduce some method to ensure that the students are learning. You could say the building of the schools is a construction project. We all know how to do that. We need experts to do that. We need a very good plan. We need a way in which we can bring those experts in in a very methodical way. And that's plan and control. Absent the IT system is going to be a little bit, it looks like plan and control. But with IT systems, there's always the question about who's going to use them. And because there's a question about who's going to use them. We know the general solution we need. We need an IT system to use in the school. But we don't really know exactly what those teachers are going to be able to do with it. So let's use an agile method. And that agile method is going to allow us to say we need an IT system, but let's build it as you move along.
[00:20:41] But then the issue of, well, how are the kids going to learn? Well, that won't be just they're more unknowns about that. So why don't we come in at the beginning and mobilize the teachers together and say to them, what do you understand as the problems of learning and then put them to work and trying to solve the problem and trying to find their own solutions. So that would be more like facilitated emergence and you could do all three in the same project. And that, I think is the key is ensuring that using the right method for the challenge that you're facing and then at the end of the day, evaluating it in the right way, too. Right. Right. First one, we could ask the question, did you build the schools? And the second one we could ask the question, is the I.T. system useful? Right. Right. And the third question, we can say are we learning how to help kids learn? It's different questions for evaluation. And if we ask those, I think it becomes less threatening too. You don't necessarily end up with this gap of 50 to 75 percent because you get information in that gap to say, here's how we progressing in that direction.
Salimah: [00:21:37] Absolutely. And for accountability, too, you can imagine like the building of the schools as the largest cost. Right. And you have all the metrics and the planning control requires to be able to say, I did not pocket that money. It actually went there, schools that got built. But there is also that area where you really don't know when you need to learn. And that doesn't often necessarily take money. It's a lot more other things. Right.
Matt: [00:21:59] Absolutely. And at the end of the day, also, you know, what we're finding is that getting back to the evaluations, we're looking at the World Bank evaluation department, they ask, is it sustainable? And they ask, will it have a development impact? The interesting thing there is that those things happen after the project. Yeah, they happen after the building teams have gone after the contractors have left. And that's about the people who are there. Right. And really, the question you asking is, do they have the capability to take it the next month? And that's where we never asked that question. Now, if you're doing facilitated emergence because you're saying we don't really know how to do that, we don't know how to facilitate sustainability. How do you get teachers to inhabit a school building in a way that makes it a school, a community of learning? Well, we want to build those capabilities that you can in a very accountable way, saying in the period of time, do we see the teachers more involved? Are they engaged? Are they learning? Are they spending more time with the kids? Now, those are things that happen when those capabilities are being built. And that's where we would have a different set of questions about what success looks like, where we don't see those being asked in pretty much anywhere at the moment.
Salimah: [00:23:07] That's really, really helpful. My last question is, you mentioned we don't teach this. What should we be teaching in schools? Because I do agree that there is a failure on all of our parts. Right. To kind of teach people that, oh, yeah, this is easy. All you need is a good technocratic solution and bingo, you're going to make change happen. And we all know that's not true. That's not how change happens.
Matt: [00:23:29] Well, you know, I mean, the first thing is that even teaching people how to do good planning control implementation would be a good start. You know, we have too many policy students who, frankly, leave our school without knowing how to write a budget. And, you know, that's a scary thing. It's so even even doing a good plan. We have a really great idea. We put a plan. I say to my students, if you really believe your plan is that good, then you need to invest in the time to understand how to get it implemented. And then you need to understand how do I mobilize finances, how do I mobilize people, how do I organize people, how do I manage politics. Right. Yeah. You can't complain. At the end of the day, I had a great idea, but the politicians let me down. No, you should earn it. You should expect it. And you should have a strategy. Absolutely. And we should be helping people to learn how to do that.
[00:24:14] There are ways of doing that. Right. And then we need to teach people that there are different ways of doing this. I think we should teach people standard project management how to teach people how to use agile tools. We should teach people how to do facilitated emergence. And in our experience, we find that when students learn these things, they are much more adept to engage with the real world. These are practical, practical tools, the tools that they can take with them when they finish their tools. We can teach people in degree programs. We can teach people in executive programs. We can teach people in the field. Now, when we don't teach this, what we are hoping for or assuming is that people are just so smart that they even know it. Or when they get into their jobs are given the space on the job to learn it. And I think we also know that that doesn't that doesn't help, because one of the things that is peculiar about public policy organizations is that they quite stressful environments and the scope for learning on the job can be less than we think because failure is not always rewarded. So I think that we need to take upon ourselves the opportunity to bring people out of that environment and teach them some of the tools that can help them to be more successful in doing this when they get into their work environments in the public policy space.
Salimah: [00:25:32] I would totally agree because even in my own experience, a lot of things that I've learned have been high stakes. You're right. When you're already on a project, it's 500 million dollars. And that's 500 million dollars that's already been signed. And there's not much leeway to be able to change things. And it would be really nice to be able to have those tools before. So you don't get to figuring out at 500 million dollars that it's not working.
Matt: [00:25:56] And certainly even if it's even if it's not 500 million dollars, but $50000, if you are doing something as a public servant, you're doing them for people who need you to do it for them. And in that situation, it's always high stakes. If you care for your job and if you came to this work of public policy because you actually care for the mission, it's always high stakes because you're working on behalf of people to solve problems that they need to have solved. I think even personally, things are always high stakes when you're working on behalf of others. When you're working on behalf of myself, I'm happy to take a risk on because I can manage that risk. But when you're working on behalf of others, it gets a little bit more scary. Yeah. And so I think people get a lot more risk averse. And we have a long literature, a very big literature that tells us that people who work in this area are more risk averse than entrepreneurs. So I think we need to equip them so that they can engage in risky areas with more confidence.
Salimah: [00:26:52] That's right. And compartmentalize the risky. Right. Do you know how to do in areas where we know how to do? And then the areas where we don't know you use these tools that will really help you learn and adapt.
Matt: [00:27:04] And I'll say the other thing that we hoping to do in some of the work, especially in the executive programs in place, is we're trying to build communities of practice as well. Because I think you can teach people, but if you can also have people who are trying to implement things better work with each other, talk to each other across the world, across boundaries, learn from each other. I think that it will help people to manage this risk as well. I'm part of a community. I don't need to know all the answers. I can learn as I go. I can click on to my community of practice and send a message and say, you know what? I'm on a project where we building school buildings. But I really want the kids to learn. Does anyone have any experience? And people will come back and they'll say, this is what we're doing. One of our goals is to build over time and we're right at the beginning. So if anyone sends a message and says, how are you doing that? I'll say we don't know we're going to facilitate its emergence. We want to build the biggest community of implementation practitioners in the world so that we can create this environment for learning. Ask us in five years how it's going. Or better than that. Join the community in five years. Hopefully you would be communicating with community members all over the world about implementation in ways that you can't do right now.
Salimah: [00:28:13] Great. Thank you so much Matt. Thanks. Always a pleasure having a podcast with you. Thank you.
[00:28:19] To learn more about the Building State Capability program, visit bsc.cid.harvard.edu.