How do citizen monitors make a difference?

That’s the key question explored by our two recent studies: an analysis of 283 projects in Kwale County, Kenya, and a two-year piece of research in northern Ghana. As well as publishing the full reports, and <a href="https://integrityaction.org/what-we-are-learning/blog/the-proof-is-in-the-product-two-new-studies-affirm-community-monitoring-s-potential-to-deliver-better-public-infrastructure/" target="_blank">this blog</a>, we’ve also created this quick quiz as an interactive way for you to explore some of the findings. [[Next->Next2]]

You can keep your own score, or not, but it’s not an exam. We made this as an easy way of sharing some serious evidence, because we know you don’t all have time to absorb 50-page reports – and because it’s fun! Of course, you can read the reports first and then come back to test how much you’ve absorbed, or just dive straight in and see how far your assumptions get you. Are you ready? * [[Yes please, let's get started!->Start]] * [[Not right now, I’d like to read the full reports.->No start]] * [[Maybe – but I’m more interested in how you’ve made this!]]

Great! First up, a bit of an explanation. The quiz is made up of a series of questions with multiple options to choose from. In each question, you can choose one option. Once you’ve clicked on your choice, we’ll tell you the answer and a bit about what the evidence said. If you want to go back at any point, you can use the arrow to your left. [[Ok, got it.]]

No problem, you can find them both on the learning page of our website <a href="https://integrityaction.org/what-we-are-learning/learning/" target="_blank">here</a>. Or read more about them <a href="https://integrityaction.org/what-we-are-learning/blog/the-proof-is-in-the-product-two-new-studies-affirm-community-monitoring-s-potential-to-deliver-better-public-infrastructure/" target="_blank">here</a>. Then come back when you’re ready, we’ll be waiting.

We can’t hold that against you. It’s made using <a href="https://twinery.org" target="_blank">Twine</a>, a free tool designed for “interactive fiction”. We’ve barely scratched its functionality, but for our purposes we found it very user-friendly and were able to do it all within a browser - though it does help to know a few HTML tags. Now, shall we get into it? * [[Sure, I’m ready now.->Start]] * [[No thanks, I’ll read the reports and come back.->No start]]

Alright, just one more thing you should know. Both studies were built around Integrity Action’s work supporting citizen monitors. Our programmes in each country were a bit different, but both involved training (extra)ordinary people to access information about projects in their local areas, gather feedback from their communities, and work with contractors and authorities to identify and resolve any issues. Now, which bit of research do you want to explore first? * [[Does citizen monitoring improve infrastructure quality?->Kenya]] A statistical comparison of monitored and unmonitored projects in Kwale County, Kenya * [[Does citizen monitoring save public money?->Ghana]] The case of monitoring infrastructure projects in Northern Ghana * [[Wait, I need to know more about the Integrity Action programmes]]

This was an analysis of data collected from 283 infrastructure projects in Kwale County, Kenya, in Sept/Oct 2022. The projects themselves were implemented between 2018 and 2022, under the departments of medical and public health, water services, and education. About three-quarters had been monitored during implementation, with the others visited for comparison. The purpose of the exercise was to gather information on the condition of the projects as of the end of 2022, and see whether there was a difference between those that had been monitored and those that hadn’t. The citizen monitors in this case were supported by our partner <a href="https://www.kygc.org/" target="_blank">KYGC</a>, through the <a href="https://integrityaction.org/what-we-do/initiatives/visibility-openness-and-integrity-through-community-engagement-2-1-1/" target="_blank">VOICE programme</a>. [[Makes sense, show me the first question.]]

This research ran for more than two years alongside the <a href="https://integrityaction.org/what-we-do/initiatives/monitoring-for-financial-savings-m4fs/" target="_blank">Monitoring for Financial Savings</a> (M4FS) programme, which was implemented in partnership with SEND Ghana. The research itself was done by INTRAC and KNUST *(info about all these organisations is on the link above)*. Citizen monitors in this programme supported the management of 15 public infrastructure projects, mainly school classrooms although there were also some market stalls and assembly buildings. Alongside other activities, monitors used Integrity Action's <a href="https://integrityaction.org/devcheck/about-us" target="_blank">DevelopmentCheck</a> (aka "DevCheck") tool to report and track problems they encountered. [[>->Ghana2]]

No problem. Well, Integrity Action is all about what we call “citizen-centred accountability”. We partner with local organisations to help citizens secure quality projects and services, like education, health, water, and essential infrastructure. In so doing, we build relationships and trust between citizens and the people who are meant to serve them, so they can identify problems and solutions together. [[Sounds good.]]

Both of the projects covered by these studies focused on citizen monitoring; you can read about this <a href="https://integrityaction.org/what-we-do/approach/citizen-monitoring/" target="_blank">here</a>, or check out <span style="color: #eca327; cursor: pointer; text-decoration: underline;" onclick="(function() {var iframe = document.createElement('iframe');iframe.src = 'https://www.youtube.com/embed/bLxS5BOC-nM?autoplay=1';iframe.style.width = '88vw';iframe.style.maxWidth = '800px';iframe.style.height = 'auto';iframe.style.aspectRatio = '16 / 9';iframe.style.position = 'fixed';iframe.style.top = '50%';iframe.style.left = '50%';iframe.style.transform = 'translate(-50%, -50%)';iframe.style.zIndex = '9999';iframe.allow = 'autoplay; encrypted-media';document.body.appendChild(iframe);var closeButton = document.createElement('button');closeButton.textContent = 'X';closeButton.style.position = 'fixed';closeButton.style.top = 'calc(50% - ' + (iframe.offsetHeight * 1 / 2) + 'px)';closeButton.style.left = 'calc(50% + ' + (iframe.offsetWidth * 0.915 / 2) + 'px)';closeButton.style.zIndex = '10000';closeButton.style.backgroundColor = '#eca327';closeButton.style.border = 'none';closeButton.style.fontSize = '2vw';closeButton.style.color = '#ffffff';closeButton.style.opacity = '0.67';closeButton.style.fontWeight = 'bold';closeButton.style.cursor = 'pointer';closeButton.addEventListener('click', function () {document.body.removeChild(iframe);document.body.removeChild(closeButton);});document.body.appendChild(closeButton);})();">this 90-second video. Once you’ve had a look at those, which bit of research do you want to check out? * [[Does citizen monitoring improve infrastructure quality?->Kenya]] A statistical comparison of monitored and unmonitored projects in Kwale County, Kenya * [[Does citizen monitoring save public money?->Ghana]] The case of monitoring infrastructure projects in Northern Ghana

No time like the present. Good luck! (text-style: "emboss")[1) Which of these statements is true?] a. Monitored projects were in better condition than unmonitored ones, [[thanks to monitors helping to spot problems early->k1a]] b. Monitored projects were in better condition than unmonitored ones, [[thanks to contractors behaving better when they know they’re being watched->k1a]] c. Monitored projects were in better condition than unmonitored ones, [[but we don’t know why->k1c]] d. [[There was no difference->k1d]] in the conditions of projects that had been monitored and those that hadn’t

Not quite! The monitored projects were more likely to be in good condition, but this study didn’t look at the reasons behind that. Early problem detection and improved contractor behaviour were noted as benefits of community monitoring in the Ghana study. [[>->K2]]

That's right! The monitored projects were more likely to be in good condition, but this study didn’t look at the reasons behind that. Early problem detection and improved contractor behaviour were noted as benefits of community monitoring in the Ghana study. [[>->K2]]

Nope, sorry! The monitored projects were more likely to be in good condition, though this study didn’t look at the reasons behind that. Early problem detection and improved contractor behaviour were noted as benefits of community monitoring in the Ghana study. [[>->K2]]

(text-style: "emboss")[2) So, monitored projects were in better use than unmonitored ones?] a. [[True->k2a]] b. [[False->k2b]]

Not exactly. There was a statistical link between projects being monitored and being in good condition, and between them being in good condition and being used well, but in a strict statistical sense there isn’t a relationship between monitoring and usage. At least, not in this data. [[>->k3]]

Yes, that's right. There was a statistical link between projects being monitored and being in good condition, and between them being in good condition and being used well, but in a strict statistical sense there isn’t a relationship between monitoring and usage. At least, not in this data. [[>->k3]]

Kwale County has an existing, constitutionally-mandated structure for project oversight, known as Project Management Committees (PMCs). These are typically made up of community leaders and other representatives. (text-style: "emboss")[3) Who was better at monitoring?] a. [[Project Management Committees->k3a]] b. [[Community monitors->k3b]] c. [[PMCs and community monitors working together->k3b]]

Not according to the evidence. A project being in better condition doesn’t necessarily mean that the monitoring was better, but what is true is that those monitored by community monitors were more likely to be in good condition than those monitored by PMCs only. This was actually true regardless of whether community monitors were working by themselves or alongside PMCs. So you hit the one wrong(ish) answer out of three, bad luck. [[>->k4]]

Yes... (sort of) A project being in better condition doesn’t necessarily mean that the monitoring was better, but it's true that those monitored by community monitors were more likely to be in good condition than those monitored by PMCs only. This was actually true regardless of whether community monitors were working by themselves or alongside PMCs. [[>->k4]]

(text-style: "emboss")[4) Ok, true or false: monitored projects were finished faster?] a. [[True->k4a]] b. [[False->k4b]]

No, sorry. There was no evidence to suggest that monitoring speeds up project delivery, and in fact it’s possible that it leads to more delays as monitors insist on things being done properly. Honestly, we believe that having a classroom built with the proper materials and foundations dug to the right depth is worth a bit of a wait. [[>->k5]]

Correct! That was false. There was no evidence to suggest that monitoring speeds up project delivery, and in fact it’s possible that it leads to more delays as monitors insist on things being done properly. Honestly, we believe that having a classroom built with the proper materials and foundations dug to the right depth is worth a bit of a wait. [[>->k5]]

(text-style: "emboss")[5) Try this one: monitored projects were less likely to be vandalised?] a. [[True->k5a]] b. [[False->k5a]] c. [[There’s not enough evidence to say->k5c]]

Hmm, you could be right, but it’s hard to say. There are indications in the study that monitored projects were vandalised less often, but it wasn’t something the data collection was designed to look at and more evidence would need to be collected. [[>->k6]]

Yes, that’s probably the most accurate answer. There are indications in the study that this monitored projects were vandalised less often, but it wasn’t something the data collection was designed to look at and more evidence would need to be collected. [[>->k6]]

(text-style: "emboss")[6) Which of these other factors had the strongest link to a project’s condition?] a. [[The year it started->k6a]] b. [[Its budget->k6b]] c. [[Whether it was an education, health, or water project->k6a]] d. [[Its location->k6d]]

No, that wasn't the finding of this study. It was the **location** of a project that was the strongest predictor of its condition, other than whether or not it had been monitored. Relationships between project condition and the sector or start date were not found in the bilateral chi-square tests, although the regressions did suggest there were some connections. Project budgets were not part of the study, so at least you dodged that one. [[>->k7]]

Well done! The location of a project was the strongest predictor of its condition, other than whether or not it had been monitored. Relationships between project condition and the sector or start date were not found in the bilateral chi-square tests, although the regressions did suggest there were some connections. Project budgets were not part of the study. [[>->k7]]

*Final question, think carefully...* (text-style: "emboss")[7) What else affected the condition of a project?] a. [[How much the local community thought it was needed->7]] b. [[The project management skills of the local authority->7]] c. [[The quality of public communication about the project->7]] d. [[The contractor’s level of expertise->7]] e. [[All of the above->7]]

Unlucky, project budgets may have been important but were not part of this study. It was the **location** of a project that was the strongest predictor of its condition, other than whether or not it had been monitored. Relationships between project condition and the sector or start date were not found in the bilateral chi-square tests, although the regressions did suggest there were some connections. [[>->k7]]

(text-style: "sway")[Probably!] Sorry to end on a trick question, the truth is that none of these factors were considered in the data collection and so any of them may have had an effect – but we’d need to do further studies to be sure. Though one thing you *can* see on <a href="https://integrityaction.org/devcheck/about-us" target="_blank">DevelopmentCheck</a> is the opinion of local communities on whether monitored projects are needed. [[>->kend]]

That’s it for the Kenya study, hope you scored well! If you need to brush up, the report’s <a href="https://integrityaction.org/what-we-are-learning/learning/learning-paper-does-citizen-monitoring-improve-infrastructure-quality/" target="_blank">here</a>. Now that you’ve seen how easy this is, want to try the questions about Ghana? * [[Sure, let’s go for it->GhanaK]] * [[No thanks, I’ve already done them->X1]] * [[No, not right now->unkeen]]

Nice, well, I guess this is it. Unless you wanted to have a go at <a href="https://media.textadventures.co.uk/games/ueim95ICo0aLD4TeJ8OYsg/index.html" target="_blank">this other quiz</a> from when we published a couple more bits of research back in 2021...? [[Yes, give me all the quizzes!->keen]] [[Afraid two’s my limit, I’m all quizzed out.->unkeen]]

You’re keen, I like you. <a href="https://media.textadventures.co.uk/games/ueim95ICo0aLD4TeJ8OYsg/index.html" target="_blank">Here’s the link again</a>. (text-style: "emboss")[Good luck!]

Not to worry, come back another day!

The research was commissioned because we believe that involving citizen monitors leads to better quality infrastructure, and hence to savings in public money – and we wanted to quantify that saving. [[I’m excited, give me the questions.]]

Here we go, happy quizzing! (text-style: "emboss")[1) What did citizen monitors tend to do when they spotted a problem?] a. [[Spoke to the contractor to fix the problem->g1a]] b. [[Reported it to the media to put pressure on the contractor->g1b]] c. [[Engaged their local community to join in advocacy efforts->g1b]] d. [[Raised it with SEND Ghana->g1b]]

Yes, this was a common one Section 3.2.1 of the report says: *“When citizen monitors were asked what actions they took when a problem had been detected during their monitoring, three main actions were highlighted. These include engaging contractors, individually or as a group, to resolve the problem, reporting to the chief and/or assembly member and making a complaint or report to a district official to help resolve the problem. Other less emphasised actions undertaken by some monitors include making a report to a media outlet, making it known to SEND or utilize the DevCheck functions.”* [[>->g2]]

Sometimes, but it wasn’t their first choice Section 3.2.1 of the report says: *“When citizen monitors were asked what actions they took when a problem had been detected during their monitoring, three main actions were highlighted. These include engaging contractors, individually or as a group, to resolve the problem, reporting to the chief and/or assembly member and making a complaint or report to a district official to help resolve the problem. Other less emphasised actions undertaken by some monitors include making a report to a media outlet, making it known to SEND or utilize the DevCheck functions.”* [[>->g2]]

(text-style: "emboss")[2) So, how did contractors tend to respond to these problems being raised?] a. [[They ignored the monitors and refused to engage->g2a]] b. [[They responded positively and fixed the problems->g2b]] c. [[They were defensive and hostile->g2c]]

That's right! The research asked 30 monitors about the extent of positive responses they tended to get from contractor; this graph from section 3.2.2 shows that most had had good experiences: <img src="https://integrityaction.org/media/o3fiarha/responsiveness.png" style="width: 100%;" alt="Graph shows 25 of 30 respondents said contractors had responded a lot"> [[>->g22]]

Not in this case (though we can see why you might have thought that!) The research asked 30 monitors about the extent of positive responses they tended to get from contractor; this graph from section 3.2.2 shows that most had had good experiences: <img src="https://integrityaction.org/media/o3fiarha/responsiveness.png" style="width: 100%;" alt="Graph shows 25 of 30 respondents said contractors had responded a lot"> [[>->g22]]

Not in this case. The research asked 30 monitors about the extent of positive responses they tended to get from contractor; this graph from section 3.2.2 shows that most had had good experiences: <img src="https://integrityaction.org/media/o3fiarha/responsiveness.png" style="width: 100%;" alt="Graph shows 25 of 30 respondents said contractors had responded a lot"> [[>->g22]]

That said, it’s true that a couple of pages later there are signs it wasn’t all rosy: *“Negative experiences reported by citizen monitors during their monitoring activities included difficulties in contacting and communicating with the contractors, which resulted in delays and lack of progress in some projects. In one instance, the monitors expressed frustration that the contractor could not be reached, and his whereabouts were unknown, leading to uncertainties about the project's status and potential abandonment. Another negative experience highlighted by the monitors was the location of a building on a waterway, which posed a significant risk and violation of construction regulations. Unfortunately, the contractor had left the site without addressing this critical issue, leaving the project in an incomplete and potentially hazardous state.”* (p32) [[>->g23]]

Incidentally, external to the research, if you have a look at the live data on <a href="https://integrityaction.org/devcheck/projects" target="_blank">DevelopmentCheck</a> (and filter it to Ghana) you’ll see that that an overwhelming majority of the problems reported by monitors have been fixed – as of November 2023, it’s up to 630 fixed out of 667 identified, which is a whopping 94%! [[>->g3]]

Alright, next question. (text-style: "emboss")[3) Which of these was **NOT** identified as an improvement associated with citizen monitoring?] a. [[Increased sense of project ownership among local communities->g3a]] b. [[Resurrection of previously-abandoned projects->g3a]] c. [[Closer adherence to project timelines->g3c]] d. [[More accountable contractor practices->g3a]] e. [[Use of better construction materials->g3a]]

Yes, you're right. Pages 24-29 of the report will tell you more about the different pathways by which citizen monitoring can add value, with further tangible examples on 30-33. As well as (a), (b), (d) and (e) from the question above, other benefits include early detection and resolution of problems, cases of reduced theft, better project design, wider and deeper public participation, and increased accessibility for people with mobility impairments. But what was **not** found was any evidence that monitoring led to closer adherence to project timelines. [[>->g4]]

No, sorry, that one *was* a benefit of citizen monitoring. Pages 24-29 of the report will tell you more about the different pathways by which citizen monitoring can add value, with further tangible examples on 30-33. As well as (a), (b), (d) and (e) from the question above, other benefits include early detection and resolution of problems, cases of reduced theft, better project design, wider and deeper public participation, and increased accessibility for people with mobility impairments. But what was **not** found was any evidence that monitoring led to closer adherence to project timelines. [[>->g4]]

The researchers looked at a handful of projects that hadn’t been included in the citizen monitoring programme. (text-style: "emboss")[4) Which projects were more likely to have been completed?] a. [[The projects that were monitored by citizens->g4a]] b. [[The projects that were not monitored->g4b]]

Sadly not according to this evidence - although that may be for other reasons. According to the project completion rates in table 5 (page 42), only five out of 12 of the monitored projects were 100% completed, while four out of five unmonitored projects had been so. In fact, five of the incomplete monitored projects weren’t even halfway done. However, what does have to be considered here is the comparability of projects. The very small sample of unmonitored projects were selected towards the end of the research – once they’d already been completed – and so their value as counterfactuals is limited. It’s also not clear if they’d ever been planned and implemented on the same timelines as the monitored projects, and certainly their funding sources were different. [[>->g42]]

Correct – although with a few healthy caveats. According to the project completion rates in table 5 (page 42), only five out of 12 of the monitored projects were 100% completed, while four out of five unmonitored projects had been so. In fact, five of the incomplete monitored projects weren’t even halfway done. However, what does have to be considered here is the comparability of projects. The very small sample of unmonitored projects were selected towards the end of the research – once they’d already been completed – and so their value as counterfactuals is limited. It’s also not clear if they’d ever been planned and implemented on the same timelines as the monitored projects, and certainly their funding sources were different. [[>->g42]]

As a wider comparison, page 14 cites previous studies on the wider Ghanaian context: *“`[Infrastructure]` projects at district levels are usually planned to be completed within 6-12 months. However, in practice, only few projects are completed during this timeline (Williams, 2018). For example, Williams (2015; 2016; 2018) found that nearly a third of projects started by Metropolitan Municipal and District Assemblies (MMDAs) are never finished. While the average project should have a 6-month duration of those that are finished, most take up to 3 years to complete.”* [[>->g5]]

(text-style: "emboss")[5) Aside from a lack of counterfactuals, what else made it difficult to compare the monitored projects with each other?] a. [[Insufficient access to data->g5a]] b. [[Varying project start dates->g5a]] c. [[Too short a timeframe->g5a]] d. [[Problems that existed before monitoring began->g5a]]

Correct! Regardless of which you chose… … That’s right, it was a trick question. Sorry about that. The truth is, all of these were limitations of the analysis; you can read about them on page 20. They don’t undermine the research’s findings, but they do mean there weren’t as many concrete findings as we may have hoped for. [[>->g6]]

(text-style: "emboss")[6) Which of these is true regarding the cost savings?] a. [[Earned Value Analysis->g6a]] found that monitored projects had a 16% smaller cost variance from their original budget than the unmonitored projects b. [[Net Present Value analysis->g6b]] found that the savings caused by monitors were outweighed by the costs of the monitoring intervention c. [[The data was inconclusive->g6c]]

Nice thinking, but no. Earned Value Analysis *was* part of the research, but unfortunately the various data limitations meant that it wasn’t possible to put a figure on the cost savings (if any) associated with citizen monitoring. [[>->g62]]

In a sense, yes – but there’s much more to it. Unfortunately, the various limitations of the research meant that it wasn’t possible to put a figure on the cost savings (if any) associated with citizen monitoring. Since citizen monitoring comes at a price, this does mean that in the very short term the cost of running the programme may be said to outweigh the strictly financial benefits (but only since any positive figure minus zero will be positive). [[>->g62]]

That's boring, but right. Unfortunately, the various limitations of the research meant that it wasn’t possible to put a figure on the cost savings (if any) associated with citizen monitoring. Since citizen monitoring comes at a price, this does mean that in the very short term the cost of running the programme may be said to outweigh the strictly financial benefits (but only since any positive figure minus zero will be positive). [[>->g62]]

The interpretation and application of Net Present Value by the researchers was also a little ambitious. “Savings” were defined as a project coming in under budget – something that was extremely unlikely in the Ghanaian context as outlined by Williams above (and was never the aim of the M4FS programme) – while any increases in budget were interpreted as a failure in monitoring, despite page 52’s note that most increases were *“due to inflation and recent price hikes in building materials”*. This basis for the analysis, coupled with lack of counterfactuals, means it was unable to explore the possible real-world effects of citizen monitoring, as illustrated below: <img src="https://integrityaction.org/media/g2qi2i1m/savings2.jpg" style="width: 100%;" alt="Visual illustrates that it would be better to define savings and losses in relation to what would have happened had monitors not been present"> [[>->g63]]

In any case, as the report’s section on Net Present Value concludes on p52, the analysis should *“be interpreted in conjunction with other factors such as social benefits, community impact, and project objectives to have a comprehensive understanding of the projects' overall value and effectiveness”.* We’ll just have to do some more learning. [[>->gend]]

And that’s all for the Ghana research! Well played, that was a roaring success. Unless it wasn’t, in which case you’re the only one who knows - so shh, don’t tell anybody. Perhaps have a read of the <a href="https://integrityaction.org/what-we-are-learning/learning/research-report-does-citizen-monitoring-save-public-money/" target="_blank">full report</a>. and come back. In any case, fancy some questions about the Kenya study now? * [[I’m in, let’s get going->KenyaG]] * [[No thanks, I’ve already done them->X1]] * [[No, not right now->unkeen]]

Cool, maybe give yourself a drink break and we'll get going. This second piece of research ran for more than two years alongside the <a href="https://integrityaction.org/what-we-do/initiatives/monitoring-for-financial-savings-m4fs/" target="_blank">Monitoring for Financial Savings</a> (M4FS) programme, which was implemented in partnership with SEND Ghana. The research itself was done by INTRAC and KNUST *(info about all these organisations is on the link above)*. Citizen monitors in this programme supported the management of 15 public infrastructure projects, mainly school classrooms although there were also some market stalls and assembly buildings. Alongside other activities, monitors used Integrity Action's <a href="https://integrityaction.org/devcheck/about-us" target="_blank">DevelopmentCheck</a> (aka "DevCheck") tool to report and track problems they encountered. [[>->Ghana2]]

Cool, maybe give yourself a drink break and we'll start when you're ready. This second study was an analysis of data collected from 283 infrastructure projects in Kwale County, Kenya, in Sept/Oct 2022. The projects themselves were implemented between 2018 and 2022, under the departments of medical and public health, water services, and education. About three-quarters had been monitored during implementation, with the others visited for comparison. The purpose of the exercise was to gather information on the condition of the projects as of the end of 2022, and see whether there was a difference between those that had been monitored and those that hadn’t. The citizen monitors in this case were supported by our partner <a href="https://www.kygc.org/" target="_blank">KYGC</a>, through the <a href="https://integrityaction.org/what-we-do/initiatives/visibility-openness-and-integrity-through-community-engagement-2-1-1/" target="_blank">VOICE programme</a>. [[Makes sense, show me the first question.]]