Abbreviated Scope of Work:

Combined Baseline and Endline Assessments for

Three USAID/BHA Awards in Ethiopia

Reports To:

Maya Rao, MERL Advisor

Coordinating with:

Needa Malik, Disaster Risk Manager

Estimated Level of Effort:

55 days

Period of Performance:

May 27 - Aug 9, 2024

Country of Assignment:

Remote

BACKGROUND

Plan International USA is a girls' rights organization. Powered by supporters, we partner with adolescent girls, children and young women to overcome oppression and gender inequality. We provide the resources that are unique to their needs and the needs of their communities, ensuring they achieve their full potential with dignity, opportunity and safety.

Plan is currently implementing two USAID/Bureau for Humanitarian (BHA) projects: one in Borena Zone of Oromia Region and one in Northern Ethiopia across the three regions or Amhara, Afar and Tigray. The project in Borena provides child protection assistance to drought-affected communities through a combination of in-kind goods and services and voucher assistance. The project in Northern Ethiopia provides multisectoral assistance to internally displaced persons (IDPs), people who have returned from displacement (returnees), and households hosting displaced and returnees (host communities). Sectors of assistance include child protection, nutrition, health, and water, sanitation and hygiene (WASH), delivered through in-kind goods and services and voucher assistance.

Both the project in Borena and Northern Ethiopia are closing out in June and September 2024 respectively, with a new award starting up in May 2024. The new award will combine the Borena and Northern Ethiopia responses under one project; however the scope will be reduced, covering a subset of woredas from the prior two awards. Child protection activities will continue only in one woreda in Borena zone while multisectoral interventions will continue under a reduced set of target woredas in Amhara and Tigray Regions (Afar is removed).

With the closeout of the current Borena and Northern Ethiopia projects, BHA M&E guidelines require a final evaluation of the Northern Ethiopia project, while the Borena project requires only an endline assessment, with indicators to be submitted in the indicator tracking table (ITT) as part of the final performance report. As the new award is starting up, Plan is also required to collect and report baseline data to BHA within the first 90 days. Given the overlap in implementation sites and most indicators maintained from the prior to new award, Plan aims to combine the prior award endline data collection and new award baseline collection into one assessment managed by one consultant in collaboration with Plan Ethiopia project teams on the ground.

SPECIFIC TASKS

* Background document review (ITT, BHA M&E guidelines, project award documents, baseline reports and terms of reference, detailed Statement of Work Annexes (see below)

* Develop work plan and sampling strategy

* Clean and analyze baseline and endline data for Borena endline, Northern Ethiopia final evaluation,

* Review and monitor the database throughout data collection and share updates and issues with field staff

* Draft and finalize final evaluation and baseline study report for Northern Ethiopia current and new awards

DELIVERABLES

* Terms of Reference with detailed sampling plan, budget and workplan for Borena endline, Northern Ethiopia final evaluation and new award baseline. Due by June 3

* Borena project data analysis and results write-up - As a final evaluation is not required for this project, a draft write-up of methodology and key findings is sufficient for internal documentation. Due by June 28.[1]

* Northern Ethiopia Final Evaluation Report - Due in early August[2]

* New Award Baseline Assessment Report - Due in early August2

* Cleaned datasets and syntax files for Borena endline, Northern Ethiopia final evaluation, New Award baseline. Due in early August.2

Roles and responsibilities of the consultant

The consultant will lead the technical aspects of the evaluation, including

* Document review

* Refinement of data collection tools and sampling strategy

* Develop/update guidance for data collection and data quality considerations

* Provide technical support in facilitation of enumerator training

* Review pilot test data and support any further adjustments needed for tools

* Review incoming data during collection and provide progress reports and data quality issues to the field teams

* Data cleaning and analysis

* Report writing

* Presentation of results in an online/hybrid validation meeting with core project team (time permitting)

Roles and responsibilities of Plan International

Plan Ethiopia will provide the following support for the final evaluation:

* Review and propose adjustments to data collection tools

* Update digital tools according to any revisions needed

* Develop handout for enumerators to support local translation of key terms

* Hire, train and manage enumerators

* Logistics for enumerator training, pilot testing and data collection

* Providing relevant and clear information to the consultant for background, planning, regular updates during data collection and debriefing for data cleaning and interpretation of results

* Prepare local government stakeholders and community representatives for facilitating access and mobilizing communities and providing guidance to the team for implementing the evaluation activities on the ground

* Review report and provide feedback to consultant for any revisions needed

* Arrange logistics and invitations for participants to results validation and dissemination meetings.

Key qualifications for the consultant

The evaluation consultant should have the following skills and experience:

* A minimum of 5 years of evaluation experience, including substantive involvement in reviews at program and/or outcome levels in related fields with international organizations.

* Familiarity with the policies, programs and operating styles of the local and national governments that are participating in the project

* Understanding and sensitivity to the diversity of cultures, traditions and languages;

* Excellent proficiency in spoken and written English.

* Advanced knowledge and experience on gender and safeguarding issues

* Experience working with communities using participatory approaches

* Solid methodological and research skills

HOW TO APPLY:

Please submit your resume/CV along with a technical offer describing your methodological approach, workplan and team composition, as well as a financial offer.

Annex1: Abbreviated Terms of Reference:

"Integrated Multi-sectoral Response to Critical Needs of IDPs and Returnees in Northern Ethiopia" (current award) and "Strengthening community resilience through addressing the critical multi-sectoral needs of returnees, IDPs and host communities in Ethiopia" (new award)

* Evaluation Purpose

The purpose of this evaluation is to assess the performance and effectiveness of the project activities undertaken throughout the current award from USAID's Bureau for Humanitarian Assistance (BHA) to achieve project outcomes and to establish baseline measurements for the new award. Endline measurement of outcome-level indicators will be integrated as part of the performance assessment. Endline results in project sites to be continued under the new award will also be considered as baseline measurements. The evaluation plan will also include data collection in new project sites under the new award to collect baseline measures in the new implementation areas. The evaluation team will also examine unintended consequences of the project interventions to inform future emergency assistance in Northern Ethiopia.

The overall objectives of this evaluation are

* To evaluate the project in terms of relevance and effectiveness, with a strong focus on comparing achievements against project targets,

* To collect baseline measurements under the new award,

* To identify key successes, gaps, and constraints to be addressed in future humanitarian programming and

* To document good practices within the current programs and to generate evidence-based lessons learned that can inform and strengthen the strategies of both ongoing and future programs.

* Evaluation Type

This is a mixed-methods, performance evaluation, primarily focused on assessing the extent to which outcomes were achieved and collecting baseline measurements for the new award. The evaluation will also examine the project's implementation processes in terms of their gender responsiveness, inclusiveness and relevance to the local context.

* Evaluation Questions

* Performance: To what extent have the activity's interventions adhered to planned implementation (e.g., schedules, participant targeting, service delivery, and outputs) and achieved intended outcomes?

* Effectiveness: To what extent has the intervention appropriately assisted women, men, boys, girls, people with disabilities and other vulnerable groups affected by the conflict in Northern Ethiopia? How did the project prioritize and engage these groups in design and implementation?

* Challenges and action taken: What were the key barriers or challenges to program implementation? How were they addressed?

* Baseline status: What are the baseline levels of key performance indicators in participating communities under the new project award?

The evaluation team will aim to highlight best practices and lessons learned under questions 2 and 3, in terms of engagement strategies that worked well and innovative solutions to barriers that could be replicated and scaled for future interventions.

* Evaluation Methods

This evaluation will use mixed methods, including project document review, participant-based surveys, direct observations, key informant interviews (KII) and discussions with male and female focus groups.

Quantitative Data Collection

The quantitative data will be collected using a household questionnaire for both child protection, WASH, health and nutrition indicators. The tool will be administered using an electronic mobile phone platform (Kobo Tool Box). The survey will include demographic questions to determine if a household has children 0-5 months, 6-23 months. If the household has a child in this age group, the enumerator will request to interview the mother of these children, as she would be best placed to provide accurate responses to questions related to indicators N08 and N09. The demographic questions will also screen for women of reproductive age. These women would then respond to the sub-questionnaire on dietary diversity to measure indicator N10. All other questions related to WASH and health indicators may be answered by the head of household or any representative of the household above the age of 18. The respondents for the child protection household questioners will be children within different age categories of 5-9, 10-14, and 15-18 years old with the questions and interview length adjusted for younger age groups. An overview of the indicators to be measured, corresponding tools and respondent type are presented in the table below. Note that one WASH indicator related to sanitation practices has been replaced under the new award with a different indicator (W13 replaced with W12).

Sampling strategy for child protection indicator CO2a: Two objectives influenced the planned sample size: achieving an acceptable margin of error (MOE) at project level but having no less than 20 observations in each stratum to make the data useful at regional level. A total sample size of 558 was thus established at the 95% confidence level, distributed over the three regions proportionally based on the size of each region's client base. Each regional sample is equally distributed over six age-sex strata, namely boys aged 5-9 years of age, girls aged 5-9 years, boys aged 10-14 years, girls aged 10-14 years, boys aged 15-18 years, and girls in this age bracket. See table below for the planned sample distribution. Borena zone of Oromia, which was previously under a separate BHA award will be integrated under the new award, with child protection activities continuing in one Woreda only. The sample size for this integrated zone for child protection baseline data collection will be finalized in discussion with the consultant.

Qualitative Data Collection

Qualitative data will be collected through focus group discussions (FGD) and key informant interviews (KII). The FGDs will be held with children and adolescents participating in child-friendly space (CFS) activities. The focus groups will be designed to measure indicator CO2.b (percent of children and adolescents participating in child-friendly space (CFS) activities who report an improved sense of safety and well-being at the end of the program). Selected HF staff, Government Bureau representatives and community leaders will serve as the key informants and be interviewed using a KII guide. The guide for each group of respondents focused on the overall performance of the project. The questions will include a critical reflection that allows respondents to mediate on both project success and challenges and to capture new knowledge and actions for the future. FGDs and KIIS will focus first on what worked well within the project and participants' suggestions for improvements. These elements will be translated into a Strengths and Opportunities, Weaknesses and Threats (SWOT) Analysis which will inform and enhance the sustainability and development of future phases of the project. For example, strengths and opportunities could be leveraged and scaled in future iterations of the project, while weaknesses and threats could be addressed through adaptations to future project designs to mitigate those challenges.

Qualitative sampling: The sampling for the FGD will be a purposive selection of children in the age groups 5-9, 10-14 and 15-18 years old who have been enrolled in CFS and attended at least 8 CFS sessions to ensure diverse perspectives from children who have actively participated in the programs. The FGDs will have 8-10 participants, with separate focus groups for girls and for boys for each of the age groups. The discussion length will be shorter for the younger groups. The KIIs will be purposefully selected participants from health facilities, project staff and local stakeholders-such as community leaders, focal points, volunteers supporting implementation-based on their involvement in child protection, WASH, health and nutrition project activities. For feasibility, the FGDs and KIIs will take place in the same kebeles sampled for the quantitative surveys.

Analysis Requirements

Quantitative data will be analyzed with MS Excel and SPSS. The analysis will include descriptive statistics and comparison of results from baseline and endline. Qualitative data will be analyzed using qualitative software, such as NVivo or Dedoose. The analysis will use codes to identify and extract key themes, and illustrative (deidentified) quotes will be integrated into the presentation of results to complement and contextualize the quantitative findings. All results will be disaggregated by sex, age group, and region. The evaluation findings will be organized and presented by evaluation question, sector, and according to the reporting format stipulated by donor guidelines.

Evaluation Timeline and Deliverables

The evaluation activities will be conducted from June 3 to August 11, 2024. The baseline report for the new award must be submitted to the donor by August 18. Therefore, the finalized version of this report must be completed a week prior, by August 11 to allow time for integrating the baseline data in the ITT and submitting through BHA's online reporting platform (AAMP).Evaluation Team Composition. This evaluation will be led by an external consultant with logistical support from Plan Ethiopia on the data collection activities on the ground.

Annex 2: Abbreviated Terms of Reference - Endline Assessment

"Provision of life-saving child protection and basic needs assistance to drought-displaced persons in Oromia Region, Ethiopia"

* Purpose

The purpose of this endline assessment is to measure the performance and effectiveness of the project activities undertaken throughout this award to achieve project outcomes. Endline measurement of outcome-level indicators will be the primary focus of this assessment. However, the assessment team will also examine unintended consequences of the project interventions to inform future emergency assistance in Oromia, Ethiopia.

* Assessment Type

This assessment will primarily use quantitative methods to measure and compare endline values of key performance indicators with baseline and target values to determine the extent to which the project achieved its intended outcomes. The assessment will also include a review of project data-such as community feedback and post-distribution monitoring data-and a small qualitative component of staff interviews and community focus group discussions to identify any unintended consequences of the project.

* Assessment Questions

* Performance: To what extent have the activity's interventions adhered to planned implementation (e.g., schedules, participant targeting, service delivery, and outputs) and achieved intended outcomes?

* Unintended consequences: What positive or negative effects did the project have beyond its intended outcomes? How could Plan mitigate any unintended negative effects in the future?

* Quantitative data collection

The quantitative data will be collected using a household questionnaire for child protection indicators. The tool will be administered using an electronic mobile phone platform (Kobo Tool Box). The respondents for the child protection household questionnaires will be caregivers for children under 15 years old and the children themselves for children above 15 years old.

Indicator

Tool

Respondents

C01: Percentage of children receiving child protection case management services who report an improved sense of well-being following case closure

Well-being questionnaire (adapted from a tool used by Plan in Lebanon)

Boys and girls (5-9yrs; 10-14yrs; 15-18yrs) receiving child protection case management services

C04: Percent of participants who report using skills learned after completing Adolescent Life Skills and Parenting sessions

Post-training questionnaire

Adolescents, parents and caregivers (10-14, 15-18, 19-29, 30-59, 60+ years) participating in life skills and parenting sessions

Sampling

Data collection for indicator C01 will be a census of all children accessing case management services under this project whose cases meet the criteria to be closed based on child protection sectoral guidelines. The questionnaires will be adjusted according to the age groups of 5-9, 10-14, and 15-18 years old, with shorter surveys and age-appropriate questions for the younger groups. Data collection for C04 will also be a census of all individuals who completed the Adolescent Life Skills and Parenting sessions, including adolescents, parents and caregivers ages 10-14, 15-18, 19-29, 30-59, 60+ years.

Qualitative data collection

Qualitative data will be collected through KIIs with male and female project staff. KII guides will include questions about positive and negative unintended consequences of the project and suggested mitigation measures for any negative effects. The KIIs will be purposefully selected project staff and local stakeholders based on their involvement in child protection activities. For feasibility, the FGDs and KIIs will take place in a convenience sample of Kebeles where project activities have taken place. The quantity of FGDs and KIIs are outlined in the table below.

Analysis Requirements

Quantitative data will be analyzed with MS Excel or SPSS. Descriptive statistics will be used to describe the basic features of the data collected. Summary results about the data collected will be presented in charts and tables corresponding to each indicator. The analysis will also focus on comparing results from baseline and endline and against targets for the life of award. Qualitative data will use codes to identify and extract key themes, and illustrative (deidentified) quotes will be integrated into the presentation of results to complement and contextualize the quantitative findings. All results will be disaggregated by sex, age group, and region. The assessment findings will organized and presented by assessment question, sector, and according to the reporting format stipulated by the BHA M&E guidelines.

* Assessment Timeline and Deliverables

The assessment will be conducted from May 27 - June 30. This award ends on June 30, so the analysis and results write-up must be completed and all invoices submitted by June 28 to facilitate award close-out.

[1] All invoices for Borena project must be submitted prior to the award end date on 30 June.

[2] Exact due dates to be discussed and agreed on with the consultant during inception meeting with Plan