Continuity of Monitoring & Evaluation Interventions during COVID-19

Introduction

In light of the current COVID-19 pandemic and the impact it has had on staff mobility and programmatic work worldwide, OIG/Evaluation has put together short guidance and materials to help ensure fluidity for evaluation and monitoring activities for on-going and new programmes.

OIG/Evaluation together with other evaluation offices from the UN Evaluation Group (UNEG) have been compiling resources and suggestions for business continuity under COVID-19 restrictions. This document is an internal guidance available on the Monitoring and Evaluation (M&E) SharePoint that will be updated as necessary. OIG/Evaluation encourages staff to share other innovative suggestions and ideas to ensure the continuation of monitoring and evaluation activities during this period. This can be done in the discussion forum of the SharePoint (for members of the SharePoint) or via email to your respective Regional Monitoring & Evaluation Officers (ROMEOs) or OIG/Evaluation (see contact details in Annex 2: Additional resources and contacts).

The suggestions and options presented here or in the annexed resource materials are meant to be applied in accordance with agreed COVID-19 protocols in the respective offices and countries. They provide alternatives mainly in case of field visits and data collection for monitoring or for evaluation cannot be undertaken or are limited due to restrictions on movement and contacts.

This Guidance Note will address the potential impacts of COVID-19 implications on planned M&E activities. It provides alternative means of data collection and approaches that may require less travel and/or are adapted to online mechanisms. The safety and wellbeing of IOM staff, consultants, beneficiaries, government partners and other stakeholders should be the paramount consideration when adapting M&E plans and activities.

Key questions to assess M&E continuation

The following key questions can help assessing the required level of adaptation of M&E activities and the ability to continue on-going M&E activities within an office or project:

Questions	Considerations
Is the project/activity/initiative on-going despite the current restrictions and if so, how does this affect M&E activities of the project?	The project/activity/intervention may be on-going yet potentially modified due to current restrictions. Monitoring therefore should also continue to ensure that we are still assessing performance however it may not be possible to fully conduct planned monitoring. For instance, to strongly limit displacement and contacts with beneficiaries, certain activities may need to be prioritized over others, and monitoring can be considered secondary given the situation. Likewise, a planned evaluation may need to be reassessed. Revisit the evaluability assessments of programmes, projects, interventions to identify the extent to which they have been impacted by changes in the activities due to COVID-19. M&E activities should thus be utilisation-focused and focusing on what is important and essential. Think about the intended use and users, including both accountability and learning purposes for IOM and for other stakeholders. Further information on how to adjust and prioritise will be addressed below.
Do M&E activities need adjustments? (What are the compliance issues?)	Ensure that staff and stakeholders are not put in harm’s way and that activities are compliant with restrictions imposed by the government. Implement any adjustments/limitations to guarantee it. Alternatives for data collection and for other evaluation options are provided below.
What are the risks and implications of any changes to M&E activities? Are there any ethical considerations too?	Alongside considering each of the above questions, an analysis should be conducted of what are the risks and short/medium term impact of dropping some parts of the monitoring or evaluation plan. This should be documented, saved to the project files, and shared and communicated to relevant stakeholders. Any change in the project’s M&E Plan should be consulted and approved by the donor if there are accountability and contractual, or financial implications.
Have you consulted with and informed stakeholders, including the donor?	Consult with relevant stakeholders and ensure that all COVID-19 implications on activities (including M&E activities) have been discussed, including with the donor. Changes to M&E activities need to be identified and reported to donors if requiring contractual amendments. Any changes should be communicated and coordinated with other stakeholders (partners, government, affected populations, local communities) so it is clear what each actor can contribute to the conduct of M&E activities with the constraints imposed.

Data collection alternatives

Monitoring and/or evaluation methodology adjustments may require data collection alternatives:

Alternative	Further details	Issues to consider
Replacing field visits with virtual information collection	• Identify focal point in community/institute that could relay potential information to the consultants/IOM Example: In the case of distribution of materials to a clinic, contacting the health worker and requesting a photograph of the materials. • Use of GIS imagery; sharing of photographs from field locations; receiving photographs of documents and mobile phone evidence/testimony from the field.	• Ensure not to be putting people in harm’s way and respecting country guidelines on COVID-19 management • Consider ethical considerations of using people from the community/institution/location as the key focal points for providing information; consider the power structures within the community, the representativeness of their perspective. “Some communities often have less access to technology and this can bias reported outcomes.”1 • Connectivity and technical infrastructure
On-line modalities for surveys and focus groups	• Use online mechanisms for survey • Conduct focus groups with people in various locations (using platforms such as Zoom, Skype, WebEx, Blue Jeans). These platform allow the session to be recorded for later transcription.	• Depends on connectivity and technical infrastructure, as well as time to respond for surveys or to organize the focus groups. • Requires collecting an providing all contact details to the evaluator and/or using local focal points to help facilitate. • Consider how the modalities may preference or exclude certain stakeholders. • In the case of recording, IOM staff should also take inform consent to everyone, including asking each participant for approval to record the session
Online Interviews	Online platforms to complete interviews (using platforms such as Zoom, Skype, WebEx, Blue Jeans). These platform allow the session to be recorded for later transcription.	• Depends on connectivity • Requires collecting and providing all contact details to evaluator • In the case of recording, IOM staff should also take inform consent to everyone, including asking each participant for approval to record the session
Prioritise document collection and review	• Focus the evaluation on evidence gathering from documentation collection and monitoring data • For monitoring, ensure stronger record keeping and photographing of processes that should be shared in real-time, and available for eventual evaluation.	• This will have limited data triangulation • For evaluation, this will also require strong monitoring data is documented and available, including contact information for stakeholders • Also consider the availability of primary data sources, particularly government officials and staff of implementing partners, both in terms of time and ICT access.

¹ https://ieg.worldbank.org/sites/g/files/tmzbdl151/files/Data/Blog/digit…

Evaluation

Evaluation planning may be affected by the ongoing COVID-19 pandemic and the diagram below provides elements to consider for on-going and planned evaluations. Country Offices should contact their respective ROMEO early on in the process to ensure that they can support in modifications, alternatives and resources.

EVALUATION PLANNING DURING COVID-19 PANDEMIC

options

Adjustments to evaluation planning and preparation:
When planning or adjusting evaluations due to the COVID-19 situation, key areas to consider are

Alternatives	Further details	Issues/risks to consider
Adjusting the Terms of Reference (ToR) to allow flexibility in the methodology and scope due to the COVID-19 pandemic	Consider adjusting timeframes and methodology, scope and budget renegotiation (depending on the scope) and adding a clause on potential disruptions due to COVID-19	• Evaluability assessment revisited • Determine essential deadlines based on the objective of the evaluation and ensure flexibility in timing and/or design due to potential disruptions. • Assess the risks associated with COVID-19 for any activities and adjustments in conformity with imposed restrictions
Strategically re/prioritize the evaluation questions and include such adjustments in the ToRs or the Inception Report depending on the evaluation stage.	• A sharper focus may be needed when drafting or reviewing the evaluation questions. Data availability and utilization should be strongly factored in as guidelines for selection and prioritization.	• Not all the purposes initially set up for an evaluation may be achievable under contingency conditions.
Identification of evaluation consultant/team	• For external and internal evaluators, you may need to favour consultants and staff based in the country (national or international) who do not have to travel. • For external and internal evaluators from outside of the country, the data collection may have to be entirely virtual with no field visit (see section on data collection above). • Self-evaluation may also be an option ensuring self-reflection, in cases where staff in the team or mission have the appropriate evaluation capacities. This should be coordinated with the Regional M&E Officer (ROMEO).	• Ensure not to be putting evaluators in harm’s way • Consider the knowledge of COVID-19 implications (health related and in terms of governmental instructions) in selecting evaluators. • Liaise with donors and relevant stakeholders to ensure that everyone is aware of limitations and agrees with decisions on evaluation consultant/team.
Evaluation design and approach adjustments	Consider the needs to adjusting the approach of the evaluation based on the restrictions in place, and in agreement with key stakeholders (see section on data collection above, and section on alternatives to evaluation below)	• Liaise with donors and relevant stakeholders is key to ensure that everyone agrees with decisions on evaluation adjustments . • List the limitations and impact on the overall evaluation quality in term

Other evaluation options

Depending on the ability to proceed and agreement to adjust the approach, here are some approaches that could be considered to still proceed with evaluation:

Alternatives	Further Details	Issues
Consider merging evaluations together	Identify evaluations on the same thematic area and organise one evaluation	• Limits interactions • Ensure that donors are in agreement and allow for that option
Consider joint evaluations with other agencies or government on similar topics	Work together to merge evaluations on similar topics	• Limits interactions • Allows to benefit from innovative approaches from all stakeholders involved • Could be a more lengthy and complex process

Alternative evaluative approaches

If a full-scale evaluation is not possible, whether a project evaluation or a merged/joint evaluation, other “evaluative approaches” can also be considered. These can help to meet M&E needs related to learning and accountability:

Alternatives	Further details	Issues
Lessons learnt e-workshop	Conduct a virtual workshop with relevant stakeholders to identify lessons learnt and identify key learnings and findings. Guidance on Lessons Learnt workshops here. See Annex 2 for further guidance on online meeting resources for facilitators.	• A consultant may organise it virtually. • Challenging to have several stakeholders together and requires connectivity. • Methodology should be designed to secure active engagement of all participants in the virtual room.
After Action Reviews or Retrospects	Conduct an action specific review to focus on elements of key interest and needs at the time. Guidance on AAR is available here See Annex 2 for further guidance on online meeting resources for facilitators.	• A consultant may organise it virtually. • Challenging to have several stakeholders together and requires connectivity.
Project performance reviews	A less robust evaluation conducted at IOM. Similar adjustments would need to be done to limit interactions. Guidance is available here	Although similar to an evaluation, the reporting process is simpler.

Monitoring

Rigorous monitoring is also often neglected when a project is facing competing priorities and deadlines or budgetary constraints. In challenging operating environments, such as in the context of COVID-19, there is therefore a need to ensure also that strong monitoring systems are in place to ensure high-quality and timely data. Once established, the monitoring system should be strengthened through exercises such as data quality assessment:

Alternatives	Further details	Issues
Data quality assessments	Document review focus only. The DQA approach focuses primarily on data collection and the use of secondary data materials to assess the quality of collection. Here is one link from USAID Learning Lab or MEASURE Evaluation with a focus on health.	• This would be particularly useful in projects with significant data collection. • Requires strong M&E systems in place

Other good practices include: develop an adapted monitoring and evaluation plan (especially for larger or more complex project), setting up monitoring tools and ensuring staff capacity to use them, review of monitoring progress in regular staff meetings (using virtual meeting and chat tools), enable real-time communication and information sharing mechanisms, and documenting and sharing monitoring results to inform project management as well as feed into any eventual evaluation or learning exercises. Besides the data collection section mentioned above, more details related to remote management and monitoring is available in Annex 1 below.

Annex 1: Draft Remote Management and Monitoring

In light of the recent situation with COVID-19, OIG Monitoring and Evaluation is making this section of the upcoming M&E guidelines available to staff as a DRAFT in order to respond to needs of continuity of programming in accordance with the procedures put in place at the country-level.

We welcome further comments, best practices or feedback to strengthen these efforts.
This section focuses on operating in fragile and complex environments, where reaching vulnerable populations in conflict-affected areas remains a challenge, in contexts with medium- to high-insecurity while maintaining the security of the organization’s personnel, as well as in the case of large and complex programmes, some with wide coverage. Remote management and monitoring strategies can help mitigate the challenges inherent to such situations and can help IOM continuously provide targeted assistance, while reducing risk to staff.

A management structure, which may have been set up as a temporary mode of operation, can rapidly become a semi- and/or permanent approach to implementation in countries with deteriorating security. While the proliferation of remote management approaches may have offered a number of recommendations to practitioners to improve their results, it has also revealed its limitations, such as the substantial implications on intervention monitoring and accountability to affected populations. This section will cover remote management as well as remote monitoring.

Remote Management

Remote management approaches have substantial implications for monitoring and accountability practices, as well as for the ability of the implementing organization to provide assurance of reaching project/programme results. Where situations may restrict staff members from meeting with beneficiaries or monitoring activities directly, they must rely on other staffing approaches or external partners. Remote management approaches are required in circumstances with problematic technical oversight, monitoring and accountability, as well as in situations with an increased risk of fraud and corruption occurring. The following are some of the common situations in which remote management approaches can be adopted:

IOM is directly operational, but with reduced staff based on ground: In this situation, while still being able to reach target beneficiaries, management is often under the full responsibility of the limited staff that are on the ground.
IOM works with an international partner organization managing projects from different locations: This is often the case when the UN imposes security restrictions while other actors, such as international NGOs, are not subject to the same constraints.
IOM works with a local partner managing the project from different locations: This approach may be particularly cost effective and beneficial, as it facilitates access to target beneficiaries while simultaneously building the capacity of the local partner.
Implementing organization fully outsources implementation to another partner: If this is the only option, it is highly recommended to closely manage this relationship to ensure a high degree of accountability.

In general, remote management, including remote monitoring and some aspects of third party monitoring, can be a temporary or a more permanent response to security and logistical challenges in the direct implementation of programs. ²Depending on the degree of remoteness, different strategies may need to be applied. IOM must identify 9
a set of general parameters and thresholds that define when direct implementation is no longer possible or advisable (e.g. what are the risks the organization is willing to take?).

Considerations include:

Physical Security/Operating Environment – Elaborate a risk and security assessment, with a:

Solid context and security environment analysis, including dynamics of conflict (UN Risk Management Unit); distance to project sites; transport types, availability and constraints; infrastructure.
Security Risk levels analysis (low/medium/high)
Access to project sites analysis: None/irregular/regular but limited.

Cost Analysis of Options – Elaborate with the following considerations:

Programme should identify how much can be invested for delegating responsibility for the implementation by causing the least impact to programme quality, including an analysis of other cost-effective options.
Possibilities of monitoring/conducting field visits; availability of information systems; identification of efficient implementing partners; guaranteeing capacity building of local partners; effective relationship with beneficiaries, for instance through community networking systems; possibilities to assess the impact of the program through remote management.

Exit Strategy – As remote management systems are often more expensive and less reliable than direct management, an exit strategy needs to be considered and regularly reviewed.

Information: IOM should also consider who makes the decision to engage in remote programming and what processes form the basis for that decision, including the legal framework.

Remote Monitoring

Remote management approaches have significant implications for organisational set-up, accountability, monitoring and the assurance of project quality. When a situation calls for remote management, the set-up and use of monitoring require more attention. Due to logistical difficulties in conducting the monitoring in complex environments, the need for additional training and contractual arrangements with a multiplication of reporting lines may arise. Strong remote monitoring approaches become key to supporting, and contributing to, remote management.

A common challenge of remote monitoring is the allocation of sufficient resources for planning and budgeting the set-up of rigorous and effective monitoring systems. It is therefore important to identify operational constraints and budgeting limitations encountered in fragile and complex environments. This ultimately may also prevent abusive use of no cost extensions. Considering such constraints and limitations in the monitoring section of each proposal may also reassure the donor on the attention paid to monitoring for guaranteeing the overall quality of an intervention.

Through the use of specific remote management approaches, monitoring of implementation can still continue. The following chart outlines key challenges in the context of remote monitoring and possible solutions to address them.

Challenge	Explanation	Suggested Solutions
Potential Deterioration in Programme Quality	Ensuring high quality in programming may be especially challenging when projects are technically complex.	Where/if possible: Preventive measures should include a clear and strong performance framework for the IP, including a solid communication plan and quality assurance mechanisms Consider developmental partner type of contract rather than pro-rata based service agreement, it forces IPs to cheat as no motivation Induction and capacity building of the IP Break down complexity into simple, digestible components. Schedule regular and reoccurring meetings with the project/program team; train staff and refresh their understanding of the subject matter, as well as conduct cross-checks and provide on-going supervision on subject matter. Increase monitoring visits, where possible, as this can contribute to remote supervision and the cross-checking of data
Weak Monitoring and Control Mechanisms	Rigorous monitoring is often neglected when a project is already facing competing priorities and deadlines or budgetary constraints. The lack of staff capacity, standardized approaches and monitoring tools, infrequent monitoring visits, low quality of collected data and the lack of information triangulation are factors that can weaken monitoring and result in poor decision-making, potential deterioration in programme quality and corruption.	Where/if possible: Ensure dedicated monitoring capacity both at the programme/project development and implementation levels. Develop a Monitoring and Evaluation plan, outlining the overall strategy for monitoring, its standardized approaches, sources, timing and management processes and procedures. Provide standardized monitoring tools that are regularly reviewed and updated. Regularly conduct capacity building of monitoring staff Introduce controls in the monitoring process, where required. Increase triangulation of information and integrate a monitoring culture among different implementing staff, beneficiaries and stakeholders. Enable real-time communication and information sharing mechanisms
Inaccurate Data and Reporting	Low quality of data can affect the quality of reporting. It can be related to limited staff capacity and/or time spent in the field due to security concerns while collecting data.	Where/if possible: Set up easy to use data collection tools, as well as a field data collection plans with options showing plan ‘A’ and ‘B’ in line with the identified possible challenges. Set up an effective data quality control mechanism at the end of every day of data collection, as well as during data entry and submission. Plan enough time for training and piloting of tools. Throughout the data collection process, call daily the field focal person to receive updates or trouble shoot challenges. This will ensure everything is on track and it can be particularly useful when working with partners. Conduct frequent data triangulation. Introduce data reporting systems that are accessible online or through cell phones and in which it is possible provide real-time feedback to bridge the physical distance.
Reduced Number of Visits and Access to Implementation Sites	Monitoring visits to implementation sites or field offices can at times be challenging. This can result in poor communication, lack of information sharing and lack of control of information, which can ultimately negatively affect the quality of implementation.	Where/if possible: Identify a monitoring team not affected by security limitations to conduct field visits on behalf of the implementing team (ensure prior training if it is an external team). Ensure consistent information flow by establishing mechanisms that provide real-time collaboration spaces, enable greater data security, seamless coordination, and improved management and control mechanisms. Ensure monitoring visits take place regularly, whenever possible. Note that the regularity and frequency of monitoring visits can often be related to the frequency of information needs, which can be assessed through the Results Monitoring Framework Favour online monitoring mechanisms as mentioned above including also receiving photographs, GPS tracking devices for distribution of goods etc. Consider the use of a third party to conduct monitoring visits of IOM’s work. These would be entities that have more access or may be less impacted by security constraints.
Limited Staff/Partner Capacity	The most common limitations are related to management, data analysis and reporting skills, as well as having a good understanding of concepts such as M&E, humanitarian action and beneficiary accountability.	Where/if possible: Introduce these concepts at the start of an intervention, during staff capacity building to ensure common understanding of requirements and expectations. Consider developing a regular internal training schedule, invest in collaborative training events with local or international actors, promote good practice presentations and use systematized tools and templates.
Weak Technical Oversight of Implementation	Providing adequate technical support through remote management can prove to be more challenging.	Where/If possible: Consider identifying a technical M&E expert within the team (field or country office). If not available, consider reaching out to relevant regional offices or colleagues at HQ to request for assistance.
Potential Weak Communication between the main Country Office or delocalized main office and offices in the field.	Communication may suffer in remote management contexts.	Where/if possible: Bring staff from head and field offices or partners together for regular meetings, either in person or virtually. Use reporting templates and real-time document sharing through online platforms, as it can support improving communication, increase accountability, follow up, and information sharing. Introduce communication protocols.
Increased Risk of Fraud and Corruption	The risks of fraud and corruption are present throughout the implementation of an intervention and may arise in remote management settings, where monitoring is weak. Fraud or corruption can occur at various levels: at the organizational level with own staff, at the beneficiary level or at an implementing partner level. Certain socio-economic and political factors can lead to increased likelihood of fraud and corruption.	Where/if possible: Take relevant contextual factors into consideration when setting up rigorous control, monitoring and supervisory systems. Incorporate sufficient checks and control mechanisms tailored to the intervention’s individual procedures and processes.

Tips: Office 365 is available for all IOM staff members and is a great tool for this.

Resource: A helpful tool for remote monitoring of interventions is the following ‘monitoring and good practice recommendation checklist’:³

³ Humanitarian Innovation Fund & Tearfund, Monitoring and accountability practices for remotely managed projects implemented in volatile operating environments, (2012), at p. 27, see: http://www.betterevaluation.org/sites/g/files/tmzbdl151/files/remote_mo….

Source: Humanitarian Innovation Fund & Tear Fund, Monitoring and Accountability Practices for Remotely Managed Projects Implemented in Volatile Operating Environments (2012), at p. 27.
Resources primarily linked to COVID19, however, this can be applied to other contexts as well: https://www.povertyactionlab.org/blog/3-20-20/best-practices-conducting…
Remote monitoring guide : http://www.betterevaluation.org/sites/g/files/tmzbdl151/files/remote_mo…

Annex 2: Additional resources and contacts
Additional resources:⁴

European Evaluation Society has put together a list of relevant articles, blogs and trainings during times of COVID19. https://www.europeanevaluation.org/evaluation-times-covid19
ILO Evaluation Office “Implications of COVID-19 on evaluations in the ILO- An internal guide on adapting to the situation” Draft March 2020.
UNDP Independent Evaluation Office “Evaluation during Crisis: COVID-19” http://web.undp.org/evaluation/media-centre/infographics/evaluation_cov…
Tips for running events and meetings : https://www.betterevaluation.org/en/blog/quick-primer-running-online-ev…
Michael Q. Patton on evaluation implications during COVID-19 pandemic: https://bluemarbleeval.org/latest/evaluation-implications-coronavirus-g…
Webinar and blog from J-PAL on conducting phone surveys: https://www.povertyactionlab.org/blog/3-20-20/best-practices-conducting…
Blog post by Zenda Ofir “Evaluation, COVID-19 and the future – Part 1”: http://zendaofir.com/evaluation-covid-19-the-future-part-1/
List of relevant resources compiled by Thomas Archibald in the links below: https://tgarchibald.wordpress.com/2020/04/01/resources-for-eval-in-a-ti…
Suggestions about data collection and design during COVID19: https://towardsdatascience.com/dealing-with-a-data-disaster-485884e857db
Discussions about evaluations and interviewing processes (including pointers for conducting phone interviews) https://www.evalforward.org/discussions/evaluation-covid19
World Bank blog on M&E During COVID-19:
https://ieg.worldbank.org/blog/bowling-dark-monitoring-and-evaluation-d…
Online meeting resources toolkit for facilitators: https://docs.google.com/document/d/1NyrEU7n6IUl5rgGiflx_dK8CrdoB2bwyyl9…
Using technologies for M&E in insecure settings: https://www.betterevaluation.org/en/theme/MandE_technology_insecure_set…
TPM during COVID - http://www.i-aps.com/pdf/Guidelines-for-Adapting-Third-Party-Monitoring…
Remote surveys : https://60decibels.com/user/pages/03.Work/_remote_survey_toolkit/60_Dec…
FGD for communities for COVID https://www.dropbox.com/sh/05ojvujpqw7qefk/AADqRm8EU5PUrJu1uJRD0o5Ka/CO… Remote Assessments of GBV risks/Focus group discussion guide for communities_COVID-19_IFRC.pdf?dl=0

⁴These resources have not been vetted by IOM but are shared in the larger M&E community in relation to COVID-19.

OIG/Evaluation:
eva@iom.int

Regional M&E Officers

DE MULLER Maite mdemuller@iom.int – RO Dakar
HARRIS Sarah sharris@iom.int – RO Vienna
HOLLINGS Jennifer jhollings@iom.int – focal point in Brussels
KIM Sokleang sokkim@iom.int – RO Bangkok
MUTIE Rogers rmutie@iom.int – RO Pretoria
NSHIMIYIMANA Theogene TNshimiyimana@iom.int – RO Cairo
REY Jhonn jrey@iom.int – RO Buenos Aires
VACA Vanesa vvaca@iom.int – RO San Jose
WAMBANDA Angeline awambanda@iom.int – RO Nairobi