As the old saying goes, if you can’t measure it, you can’t improve it. But of course, what you measure is important too. In this episode of The Development Podcast, we explore why the World Bank Group is placing impact center stage with a slimmed-down and focused scorecard.
We’ll hear how the scorecard will measure success, as well as identify areas for improvement. We’ll get to the heart of what impact measurement is, why it matters and how it has gained traction across the development community. We also get snapshots from two projects: one in Mozambique and one in Jordan. Listen now!
Tell us what you think of our podcast here >>>. We would love to hear from you!
Featured voices
- Ed Mountfield, Vice President, Operations Policy and Country Services, World Bank
- Lu Shen, Director, Results Management and Aid Effectiveness Division Strategy, Policy, and Partnerships Department, Asian Development Bank
- Beneficiaries: Voices from development projects in Mozambique and Jordan.
Transcript
[00:00] Samuel Kwadwo Owusu Baafi: Hello, and welcome to The Development Podcast from the World Bank Group. I'm Samuel Owusu Baafi in Washington DC. Today's episode is all about achieving impact in development. We're asking what it means, why it matters, and how it is measured. We'll get a snapshot from a World Bank project in Mozambique, where climate-resilient classrooms are providing children with a safe and secure environment to learn.
Caterina: When it rained, we didn't even sit in a classroom. We put our books in plastic sheets, and we ran home. Now, the school is excellent.
Samuel Kwadwo Owusu Baafi: We'll do a deep dive into the World Bank Group's approach to measuring impact through its new scorecard.
Ed Mountfield: It's about shifting from dollars to lives transformed. Going under the bonnet a little bit, and recognizing there's good news, and there's areas where we need to double down.
Samuel Kwadwo Owusu Baafi: And we'll get the view from the Asian Development Bank on why impact is so crucial in development.
Lu Shen: It's about improving people's lives, at the end of the day.
Samuel Kwadwo Owusu Baafi: All that and more, coming up in The Development Podcast.
Keeping score matters. Think of a sports game, a bingo hall, or even a political debate. Throughout our working lives, we are appraised. And many of us will still remember what it felt like to get our school report detailing our strengths, areas to be improved, and maybe some unexpected achievements. When it comes to measuring impact in development, the principle stays the same; keeping score is critical. We'll be unpacking the World Bank Group's new scorecard, and exploring how it'll steer the organization's approach to impact. But first, let's learn from a couple of projects that really help explain what we mean by impact. Our producer, Sarah Treanor, found out.
[01:57] Sarah Treanor: The sound of children at their school in Mozambique. These children, like many others, can find their education disrupted during extreme weather, so things like cyclones and floods. In 2023, Cyclone Freddy hit the region, causing widespread devastation. But climate-resilient classrooms retrofitted with reinforced structures can help keep schools safe and open. Let's hear now from Caterina, a ten-year-old pupil, about the changes.
Caterina: When it rained, we didn't even sit in a classroom. We put our books in plastic sheets, and we ran home.
Sarah Treanor: Augustino Mabriga, Director of Rosina Muckle Primary School, says this...
Augustino Mabriga: There was so much water coming in, that the children sometimes had to flee their classrooms.
Sarah Treanor: The Resilient Classroom Initiative, supported by the World Bank, has so far transformed more than 700 classrooms in Mozambique. Around 1.2 million people have directly benefited from this project. Over 58,000 children have gained access to improved education infrastructure. In Caterina's school, this meant reinforcing the roof structure, removing the old one, and replacing it with a roof able to withstand extreme weather and prevent the school from flooding. When Cyclone Freddy hit the country last year, the school remained dry and unharmed. For children like Caterina, it means they don't have to worry about a disrupted education and can look to their own bright futures.
Caterina: Now the school is excellent. I really like learning. I'd like to be a teacher when I grow up.
Sarah Treanor: Now let's travel to Jordan.
Nada El-Ramahey: I benefited from the supplementary support, in that it covered my children's needs. Before receiving the supplementary support, many things were lacking for my children, like paying rent, school expenses, water and electricity. Now, thank God, it covers my needs and those of my children.
Sarah Treanor: Nada El-Ramahey is just one of the 1.2 million people affected by COVID-19 that the National Aid Fund's Cash Transfer Program, supported by the World Bank, has reached. Her experience echoes that of Zaher Zayadna. He is training in solar maintenance. Part of the fund's aim is to provide support for people gaining skills, moving from the informal to the formal sector.
Zaher Zayadna: I registered for the program through my training in solar energy at the National Aid Fund. They told us about the opportunity and to better my living condition through the monthly payments. It helped prepare me to find jobs. They gave me a certificate for working in this field so I can rely on myself and improve my family's living condition.
Sarah Treanor: The program is also a model in terms of digitized and transparent practices. It's had a big impact. According to analysis from the World Bank, Jordan's Unified Cash Transfer Program in 2021, for example, is estimated to have reduced inequality by 0.7% and overall poverty by 1.4%. Jordan's National Aid Fund Cash Transfer program has provided a lifeline of cash support for the poorest, most vulnerable households. It's now the largest program of its type in the region. Fadia El-Henawey has also gained new skills and is training to be a chocolate maker. She tells us more.
Fadia El-Hanaoui: I've received training in chocolate making, how to produce it and market it. When we receive training at the National Aid Fund, it helps us with finding job opportunities. Because the support will not be provided for the long term, my dream is to establish my own business in the future.
[06:51] Samuel Kwadwo Owusu Baafi: Those are some pretty incredible personal stories. And in a way, they get to the heart of what the new scorecard is; measuring impact on people. As we know, the world faces many crises, from food insecurity, climate change, inequality, conflict and biodiversity loss. The World Bank Group is working to address these global crises with a mission of creating a world free of poverty on a livable planet. But how does the bank, its partners, shareholders, and really anyone with an interest in or involvement in the institution know how it's going? How do we know we are on track? It's a great question. Well, earlier this year, a revamped system for keeping score and measuring progress was introduced. The new scorecard streamlined the number of metrics it uses from over 150 indicators to just 22. It also moves towards radical transparency and measures impact across all different parts of the organization. The scorecard is kind of like the World Bank Group's report card. I wanted to learn more, so I asked Ed Mountfield, the Vice President of Operations Policy and Country Services to join me in the DC studio to explain how it works. Hi, Ed.
Ed Mountfield: Hi, Sam.
Samuel Kwadwo Owusu Baafi: It's a pleasure to have you here with us today on The Development Podcast.
Ed Mountfield: Great to join the podcast.
Samuel Kwadwo Owusu Baafi: The scorecard. We're just going to dive straight in. Can you tell us what exactly is the World Bank Group scorecard?
Ed Mountfield: So the World Bank Group scorecard is our yardstick for measuring the results of the development work that we do all over the world. We have a new mission for the World Bank to end poverty and boost shared prosperity on a livable planet. And the scorecard is about really translating that into the key development outcomes that are necessary to achieve that bigger picture goal. And it's also about helping us be more accountable and ultimately to focus and strengthen the impact of our work. So the central piece of the scorecard is focused on monitoring our progress across 22 results indicators that we think are absolutely mission-critical to getting to where we need to be, to help people all over the world to deal with the climate emergency and to uplift their lives. And the scorecard provides a clear view of how we're doing on some of the most critical global challenges, from protecting the poorest to adapting to climate change, achieving gender equality, and ending learning poverty. So it's really about focusing on results, focusing our work around achieving those results and holding ourselves accountable. The other thing that's a big step forward with this scorecard is that it's much more transparent and systematic than it's been in the past. So we've really worked to clarify the methodologies by which we measure our impacts to actually document them, publish those methodologies, and also to make the data itself much more transparent so you cannot disaggregate it, you can see all of the backend data. And so we are not where we want to be yet. It's a journey, but we are working towards radical transparency as part of this scorecard.
[09:53] Samuel Kwadwo Owusu Baafi: I can only see that being a good thing. And what exactly is the scorecard designed to measure? And how is it different from the previous iteration?
Ed Mountfield: An insight from previous scorecards we've had in the past is that we probably tried to measure and target too many things, and if you make everything a priority, kind of make nothing a priority. So really, zeroing in on the things that we think are really the most impactful things. So it's designed to measure the outcomes of the support that we provide to developing countries. We use a simplified set of indicators that are trying to emphasize quality as well as usage. And these indicators have been carefully selected through a broad consultation as part of our World Bank Group evolution process with the shareholders and partners. There are all sorts of insights that come from the data that are really starting to transform the way that we work. So just to give one example, a large part of the world's population currently faces moderate or severe food insecurity. In fact, over 60% of people in low income countries are in that category. So our current efforts are responding to this challenge with total results from our active portfolio of projects expected to reach nearly a third of a billion people and counting. And we're working to build that portfolio up and to ensure that it's delivering. Our support is aiming to target the most at-risk populations, the majority of results reaching those in Africa and South Asia. And food insecurity will continue to remain a challenge. So we can use this to target our work to hold ourselves accountable, but also to raise questions like, "Is that enough?" We've still got a huge gap to goal, so how can we take it to the next level? And what can we learn from the experience? Also, the disaggregation of the data is something that's radically new. So in the past, we've kind of published the headline numbers, but being able to drill that down and say, "Okay, this is what we're achieving in terms of tackling food insecurity." But how much of it is in Africa? How much is in the low income countries? Are we actually tackling the problem where the problem is most intense? Or do we need to rebalance some of our work and shift our focus? So that's, I think, a big part of the new approach.
Samuel Kwadwo Owusu Baafi: It sounds like this approach really puts people at the center of the World Bank's work.
Ed Mountfield: Yeah.
Samuel Kwadwo Owusu Baafi: And in that same vein, I think transparency is also very important. So how will the results be published?
Ed Mountfield: It's a work in progress. At the moment, we're covering only a portion of the results in our current portfolio. So as we work to make sure that we're capturing the results we achieve using these methodologies across all of our projects, we'll build these results up over time and reveal, I think, more of the impact that we're having. We've made a lot of progress and during the World Bank IMF Annual Meetings next month, the next set of the scorecards indicators will be released. We'll releasing 20 of the 22 indicators on a new interactive and dynamic website, which will make the data accessible and reproducible and disaggregable for everyone. So all of the indicators where possible will be disaggregated by gender, by youth, by disability, by country, region. There's going to be a strong focus on the development of people in client countries. We know that data is very powerful, but it's only one way of telling the story of the impact that you have. So we're also going to include in the scorecard what we're calling results narratives. So these are a series of analytical notes that draw on the data, but also tell the story of how we achieve the results that we achieve and what some of the challenges are, and getting a little bit beyond the data to tell a bigger picture story.
Samuel Kwadwo Owusu Baafi: For somebody who works in communications, this is music to my ears. Showing what that impact really means on the ground to the people that we support. We're going to cap this conversation off with a question that I think you've touched on in everything we've talked about. I want us to really just highlight this. Why is keeping score important?
Ed Mountfield: The World Bank Group is ultimately not about the money it commits or disperses, it's about the difference it makes to people's lives on the ground. And that's what motivates all of us to do the jobs that we do all over the world. So I think it's about focusing on the thing that really matters, which is the impact we have on people's lives. So I think that's ultimately what it's about. It's about shifting from dollars to lives transformed. It's about shifting from just telling the headlines to going under the bonnet a little bit and recognizing there's good news and there's areas where we need to double down.
Samuel Kwadwo Owusu Baafi: I love that. It's a shift away from dollars to lives transformed. Ed Mountfield, it was a pleasure chatting with you and I look forward to a lot more conversation coming up.
Ed Mountfield: Great. Enjoyed it. Thanks a lot, Sam.
[14:46] Samuel Kwadwo Owusu Baafi: All right. So we've heard lots about why and how the World Bank is honing in on measuring impact. And in a way, it's part of a sea change throughout the development community. Lu Shen is the Director of the Results Management and Aid Effectiveness Division at the Asian Development Bank. So in short, she knows a whole lot about measuring impact. So someone who's coming into this topic cold, no idea what we're talking about, impact, development, etcetera, someone who does not know what we do, how would you explain impact and why it matters?
Lu Shen: It's about improving people's lives, at the end of the day. We talk about results all the time and we have different metrics and we have different targets, but at the end of the day, what is the purpose of anything that we do? I think back to when I joined ADB, one of the cornerstones of ADB is alleviation of poverty. And so I think this is at the fundamental basis for everything that we do. How do we and our work improve people's lives? I think that's really at the core of all of the MDBs.
Samuel Kwadwo Owusu Baafi: What happens when we miss targets? Because a lot of time we set targets, but one that always jumps to mind are the SDGs. What are your thoughts on that?
Lu Shen: Yeah, this is a tough one because I think it's important to set targets because it gives people something to work towards. But I think there are a lot of different things in the pathway that is challenging. When you set targets, there is the whole idea of what is your target? How do you go about measuring it? And I think these are really tough things to do because when you think about it conceptually, it's one thing, but ultimately, in order to have any kind of impactful reporting, you need to have the same metrics. People need to measure it in the same way. And everybody has to buy into the same system. And I think that the challenge with working with our developing member countries is that everybody has their own way of doing it. That policy formulation is great, it's very academic and it's very idealistic. Target setting is important, but it's much more important to think through what are some of the things that we can do to make that consistency in terms of developing metrics and developing tools for people so that they can report in the same manner. When you do come to the end, for example, the SDGs by 2030, what are some of the achievements that we have versus what are some of the gaps?
[17:36] Samuel Kwadwo Owusu Baafi: Yeah, I think that's a very pragmatic way to think about it. Right? I think people listening would be very interested in hearing an example of a project that you think has had a fundamental impact, whether it's short term or long term, on people's lives in Asia.
Lu Shen: I think one good example is there was a wash program. This was actually a unique project because they were looking at behavioral changes through a water sanitation project, and so about hand washing for kids. And so there's access to clean water, that's one thing, but if you actually don't wash your hands, ultimately, how's that going to help improve your health? So I think this is a really, really interesting project because it really delves into the human factors of how do you actually get people to change behavior. When you do mobilize, for example, civil society, when you actually have the right conversations with the government as well as communities to make that kind of incremental changes so that they can actually start with a different behavioral pattern, that's coming in this case from the children in schools. And so there's that kind of ripple effect that we actually did not expect. Ultimately, when you do this, the correlation about how often do you fall sick? And people will have that kind of realization like, "Oh, well, maybe there's something to it."
Samuel Kwadwo Owusu Baafi: This has been super illuminating, Lu Shen, but before I let you go, there's one thing I wanted to touch on. Now, the World Bank has its new scorecard, but can you elaborate a little bit on how important it is that there's alignment in measuring impact across all the MDBs?
Lu Shen: I think it's extremely important for us to put our heads together to come up with a common methodology. And I know that the climate team has been working on this since the beginning of the year because we want to be credible in what we measure, especially when it comes to things like climate where we can have the same methodology, we can use the same tools to jointly report on what is our collective contributions. And this is incredibly important. It's not easy, just because we all know outcomes are difficult, because we talk about attribution versus contribution all the time. It's tough.
Samuel Kwadwo Owusu Baafi: And that's about all for this episode of The Development Podcast. Join us for the World Bank Group IMF Annual Meetings at the end of October to learn all about the new scorecard results. And we hope you really enjoyed this dive into the idea of impact and keeping score. It's a lot to think about. Be sure to tune in next time. And if you'd like to get in touch, send us an email at thedevelopmentpodcast@worldbank.org. Many thanks for listening. It's Samuel, and we'll be back soon.
ABOUT THE DEVELOPMENT PODCAST
This international development podcast brings together the data, research—and solutions—that can pave the way to a sustainable future. Through conversations focused on revealing the latest data, the best research, and cutting-edge solutions, let us introduce you to the folks working to make the world a better place. Don't miss an episode! Listen and subscribe for free on your favorite platform. And rate our show! ;)
Tell us what you think of our podcast here >>>. We would love to hear from you!
ABOUT THE WORLD BANK
The World Bank is one of the world’s largest sources of funding and knowledge for low-income countries. Its five institutions share a commitment to reducing poverty, increasing shared prosperity, and promoting sustainable development on a livable planet.