By: Korinne Chiu, Director of Monitoring and Evaluation
What does success look like? This is the million dollar question I like to ask when beginning to design an evaluation.
Being able to quickly pull up a child’s vaccination history with the scan of a fingerprint or a QR code?
Health worker comfort with using the VaxTrac system?
Reducing the time it takes to receive district-level records of immunizations?
Reducing vaccine waste in a region?
The million dollar answer depends on who you ask, and there’s usually more than one answer. This is why it is important to identify stakeholders or groups of people involved in an intervention and ask each group this question. Each stakeholder group brings different resources to the table and may have a different idea of what success looks like to them. Some of our stakeholders include health workers, ministries of health, funders, and field staff.
As the Learn team, it is our job to talk with different stakeholders to understand what outcomes stakeholders would like to achieve and the process for doing so. Talking to different groups increases buy-in and understanding for all parties involved and helps us come to consensus on goals we are trying to achieve.
Some of the ways that we have worked with our stakeholders to understand what success means to them includes engaging in discussions about what results they would like to see at the end of a project. From there, we worked backwards to understand what activities would need to be conducted and what resources would be needed to conduct those activities.
We worked on answering these questions with staff in Benin while reviewing a draft of our logic model. Through a facilitated discussion, we gathered information from staff who are in the field on a daily basis to understand what they view as the main objectives of VaxTrac’s work and how they would define success. In preparation for this meeting, Learn Team drafted key outcomes and indicators that each stakeholder may be interested based on their daily activities. Staff were asked to brainstorm resources, activities, and outcomes. After staff brainstormed their list, we compared the list to the drafted logic model to see where there were areas of overlap and where there were things that were missing from the drafted logic model. We came away with a more comprehensive document that reflected the team’s daily work.
For example, one outcome that staff noted as a result of using VaxTrac was that data were more secure in a cloud-based system when compared with paper records for a number of reasons. Reasons included that fewer people had access to the information and that information stored electronically was more resilient than paper-based records. Learn team did not have this outcome listed previously and incorporated it into the logic model. Having a visual representation of our activities, outputs, and expected outcomes helps us focus on what each stakeholder wants to achieve as well as understanding secondary outcomes or benefits.
After we collect information from our different stakeholders, we group outcomes into themes and prioritize the types of information we will collect. Learn team hones in on what the priority questions are and what information is feasible to collect in a given time period. We figure out the best way to collect information and when to collect it. For example, we may conduct a focus group with health workers to understand how comfortable they feel with using the system or we may use data that is automatically collected on the tablet to understand how long it takes to retrieve a record.
Asking what success looks like during the planning phases helps us stay on track to document our progress and learn as we go. It is a great way to understand different perspectives and to prioritize data collection. It gets people thinking and acting with an end goal in mind.
Some of our outcomes that we are trying to achieve and monitor on a consistent basis include:
- System use- health workers are consistently using the VaxTrac system in their daily work
- Data quality- data entered into VaxTrac is consistent with paper records and vaccination schedule requirements
- Data accessibility- data is available in a timely manner. In practice, this would measure the amount of time between completing an electronic form and sending the form to the cloud, where decision-makers have access to the data.
- Data use- health workers and health officials have the capacity to use the registry system reporting and data to monitor clinic or zone progress and make data-informed decisions related to childhood immunizations.
Since we are often constrained by resources such as time, money, and personnel, we must focus on which information to collect that will best provide data on the outcomes we desire. This information tells us if we implemented the project as we intended and are achieving the results we set out to achieve. For example, regarding the outcome of system use, we can track health worker activity to see if a tablet is being used on a vaccination day as well as how many children were registered on a given day.
So, maybe the next time you start a project you’ll ask yourself, “What does success look like for me?” Once you define success, the next step is figuring out what evidence you need to support this and how you will gather this information.