Final Evaluation of the Sustainable Mixed Agriculture Livestock Livelihoods

The Sustainable Mixed Agriculture Livestock Livelihoods (SMALL) project is an initiative of Heifer International Indonesia that aims to help needy farm families in Sumatra. After being implemented for seven years, two years longer than it was originally planned, the project is about to end. The original proposal recommends that a final evaluation should be carried out at the end of the project to look at the impacts of the project on both the project partners (PPs) and self-help groups (SHGs), and on Heifer International Indonesia.

The rationale is that the evaluation should be used as a learning vehicle to look at the strengths and concerns of the project for better implementation in the future.

To do this, Heifer ID assigned us to facilitate the evaluation.

Methodology

The evaluation was carried out in three stages as follows:

1. Desk study of the documents of the SMALL project;
2. Field visits to respective PPs and associated SHGs to collect data, including a visit to Heifer International Indonesia headquarters in Medan for a briefing about the project. In addition, the consultants also carried out direct field observation to check the presence of physical outputs of interventions;
3. Data analysis using, among other, triangulation methods (Weiss, 1998: 263).

The briefing at Heifer ID gave us details about how the project was designed and implemented, and how engagement with PPs and SHGs was conducted.

Because the area covered by the project is so vast, basically the whole of Sumatra, and the starting date of project implementation by each PP was different as well, it was decided to randomly select five PPs using the so-called “lottery draw technique” . These PPs are dispersed in four provinces of Sumatra where Heifer ID works, and each of them has a different start date/year of project implementation. They are:

1. KEMASDA in South Sumatra, where project implementation began on October 1, 2001, and ended in 2003;
2. YBTI in North Sumatra, where the project began in August 1, 2003, and finished in 2006;
3. TOTALITAS in West Sumatra. This PP started its project on July 19, 2005, and just ended the project in June 30, 2008;
4. YABIMA in Lampung, which commenced its project on January 20, 2006. The project is ongoing and will end late December 2008 or early January 2009; and
5. PANSU in North Sumatra. The project of this PP began in March 6, 2006, and is ongoing. The project is planned to end in February 2009.

Focus group discussion (FGD) was used to collect relevant data and information from respective PPs and SHGs. The list of questions used to direct FGDs was derived from the questions listed in the Scope of Work (SoW) with additional questions created by the consultants to assess the beneficiaries’ level of satisfaction with particular aspects such as post-project income, assets, quality of life, social capital, group cohesiveness, participation, role of women, influence of certain groups in the community, project relevance to beneficiaries’ needs, services provided by the project holder, and appropriateness of support given by PP to meet the beneficiaries’ needs. Level of perceived satisfaction and changes was assessed using a rating technique.

Since various approaches (i.e. desk study of documents, direct field observation, and interviews or FGDs) were used to collect data, triangulation methods were used to cross-check those data (Weiss, 1998: 263).

Identification and assessment of impacts employed the Hoofprint Model recommended by Chianca (2006). According to this model, impacts of activities generated by the presence of Heifer International are expected to occur at three different levels, i.e. above the ground, on the ground, and below the ground.

The model uses the metaphor of a footprint – the impact on the environment left by a traveling creature. In this case, the creature is, of course, a heifer, and its conventional impact is the footprints it leaves on the ground, i.e. apparent changes that occur at the beneficiaries’ level such as, among other, income, herd ownership, education and health. Since the effects of moving animals are not confined to on the ground, there are two other dimensions to the model, i.e. (1) tracks left above the ground, corresponding to the broken brush and branches that mark the heifer’s path (impacts in the communities, region, etc.); and (2) impacts below the ground, representing the desired changes in fundamental values, especially increased sense of loyalty to the group rather than just to oneself and one’s family, and increased social capital.

In practice, on the ground impacts were assessed from changes that occur at the beneficiaries’ level (e.g. improvement in family nutrition and/or income, herd ownership, and children’s education – achievement of SMALL project goal number 4 “improve family income and/or nutrition”; livestock care and management – achievement of goal number 1 “strengthen self-help groups”; and environment care & management – achievement of goal number 5 “protect/improve the environment”).

Above the ground impacts were assessed via changes that occur at community level and beyond (e.g. adoption of Heifer practices by non-Heifer families). Included in this level assessment were (a) empowerment at the community level, and (b) system and policy improvement.

Below the ground impacts were assessed through achievement to date of the Passing on the Gift scheme (i.e. passing on more and better animals than received – achievement of SMALL project goal number 3 “self-help groups will practice values-based activities”).

Other evaluation components such as relevance, efficacy/effectiveness, efficiency and sustainability were approached using questions recommended in the Scope of Work and guided by work of Dale (1998) and Clarke (2000). So with other components, i.e. institutional impact & capacity building (achievement of SMALL project goal number 2 “strengthen local NGOs/project partners”), and gender equity, which were guided by works of CIIR/ICD (2000) and Neimanis (2005), respectively.