Preface
Executive Summary
CIRAC Response
Tabular summaries of the 13 Evaluations
PrefaceThis is the first in a series of reports being produced by the International Reflect Circle, CIRAC.
The International Reflect Circle (CIRAC) was set up in March 2000 as a democratic space for Reflect practitioners from diverse organisations across Africa, Asia, Latin America and Europe. This circle seeks to promote solidarity of Reflect practitioners at different levels around the world in order to strengthen international exchange and learning, and build a wider movement. CIRAC aims to be inclusive of all serious Reflect practitioners and has links with over 350 organisations using the approach in 60 countries. CIRAC communicates with these diverse practitioners through its publications and practical resources, through an e-mail network and a new website (www.reflect-action.org), as well as through feedback systems from meetings and linkages to training and exchange workshops around the world. CIRAC is co-ordinated each year by 2 people nominated from each region and holds annual meetings (most recently in South Africa in May 2001) with balanced representation from diverse organisations and networks around the world.
Forthcoming publications / resources from CIRAC include:
· Global Survey of Reflect Practitioners;
· Approaches to Training in Reflect;
· Participatory Monitoring and Evaluation in Reflect;
· Power Analysis and Processes of Transformation in Reflect;
· Practical Resource Materials on Communication and Power.This first publication is an attempt to consolidate learning from 13 different external evaluations of Reflect that took place mostly in 2000. Each of these evaluations was commissioned independently to serve different purposes for different organisations working in different countries. In each case there were different evaluators brought in with different biases and different approaches. The task of consolidating these into a single document was formidable and we thank Abby Riddell for her remarkable achievement in having done so.
In the spirit of Reflect, we have circulated Abby's consolidated report to practitioners (through CIRAC). A team of practitioners from Ghana, Mozambique, Bangladesh, Pakistan and the UK, who are also working on wider learning around participatory monitoring and evaluation, met to review and discuss different responses - and have compiled these into a consolidated set of comments. Rather than use these to re-edit the report we are publishing these consolidated comments, which are appended here, after Abby's original report. We hope that this dialogue between evaluators and practitioners will continue, as there is much that can be learnt from such an exchange. Enjoy your reading, and let us know if you have any comments!
David Archer - June 2001 (on behalf of CIRAC)
Reflect (Regenerated Freirean Literacy Through Empowering Community Techniques) was developed as a radical new approach to adult literacy between 1993 and 1995 through field practice in Uganda, Bangladesh and El Salvador. Since then it has spread rapidly so that in 2000 there are over 250 organisations working with Reflect in a diversity of programmes in over 50 countries, led by national and international NGOs, local and district governments, community based organisations and social movements.
Through 1999 and early 2000 a series of external evaluations were carried out which looked at the outcomes and impact from Reflect programmes in several different countries. This review is of the 13 evaluations completed to date of Reflect in 11 countries.
The purposes of this review are threefold: 1) to synthesise and summarise the 13 evaluations, identifying trends in literacy and empowerment outcomes; 2) to identify and classify trends in approaches to evaluation and the indicators used; and 3) to draw out some key hypotheses to frame further research / evaluations.
The report is organised as follows. Section 2 presents individual summaries of each of the 13 evaluations, in alphabetical order by country. At the end of the section there is a table of each evaluation, summarising the following:
· Programme context
· Reflect approach
· Evaluation purpose and timing
· Evaluation methodology
· Literacy and numeracy outcomes
· Empowerment outcomes
· Other outcomes
· Weaknesses
· RecommendationsSection 3 outlines several common issues / lessons that arise from the Reflect programmes. Section 4 gives an overview and lessons from the evaluations themselves. Section 5 details some of the indicators used in the various evaluations. Section 6 follows on with guidelines for establishing monitoring and evaluation frameworks for Reflect programmes. Finally, Section 7 puts forward several recommendations and proposals for further work.
Lessons / Issues
The most fundamental issue that was raised in a number of the evaluations was the relative priority given to literacy or empowerment in Reflect. Not all groups in-situ will have fully embraced the 'renewed' definition of Reflect:"…a structured participatory learning process, which facilitates people's critical analysis of their environment, placing empowerment at the heart of sustainable and equitable development. Through the creation of democratic spaces and the construction and interpretation of locally generated texts, people build their own multi-dimensional analysis of local and global reality, challenging dominant development paradigms and redefining power relationships in both public and private spheres. Based on ongoing processes of reflection and action, people empower themselves to work for a more just and equitable society." (Phnuyal, Archer and Cottingham, 1998)
Together with often ill-defined outcomes, this has made evaluation particularly difficult.
The 'renewed' definition is full of the ideals of potential achievements of Reflect programmes. It presents an evaluation challenge that goes considerably beyond testing the achievement of basic literacy and numeracy. It also crosses several disciplinary boundaries that package different types of outcomes such as literacy and numeracy, empowerment, development, democratisation, poverty alleviation, etc.
This much broader definition comes at a time when some of the international development agencies are reconsidering their earlier abandonment of adult literacy programmes on grounds of inefficiency and ineffectiveness. Thus, the evaluations of programmes such as Reflect are receiving close scrutiny. However, some of these disciplinary boundaries are likely to continue to pose challenges, especially where these boundaries are reflected in ministerial divisions such as between education and community development.
Related to this fundamental intertwining of literacy and empowerment is the often missing relationship between Reflect circles and other community development organisations. For the ideals of the 'renewed' definition of Reflect to be realised, there needs to be a wider understanding of the development and empowerment aims and an integration of Reflect practice beyond those using the approach.
In addition, there is the often-missing relationship within international development agencies between Reflect programmes and their other sectoral work.
Several of the evaluations suggested that facilitators be offered additional training which could help them to extend the development aims of the circles, for instance, such things as micro-planning, strategic planning, advocacy, leadership training, etc. Clearly, where the aim is for the circles to be well-resourced, well-functioning community development organisations, utilising the literacy that has helped to empower them, it is important that the necessary skills to further such aims are available.
Related to all of the above points is the need to tackle the necessary resource requirements for action points. This may mean leveraging resources from governmental and non-governmental organisations alike, or it may entail attracting new resources. Especially in those circles within the most marginalized communities, it is important that they are not left high and dry just at the point at which they are mobilising themselves. This is an issue raised in several of the evaluations and underlies the challenge of promoting literacy in a context of empowerment, particularly where the support structures are not inter-sectoral, but within, say, a non-formal education department alone.
There is an inherent tension between a successful Reflect programme and its being scaled up with large state involvement. The role of the state was an issue in several of the evaluations, not only in terms of contrasting state-run literacy programmes with Reflect but also in terms of establishing partnerships between different stakeholders.
Facilitators are the crux of Reflect programmes. Thus, it is no surprise that there were many issues raised in the different evaluations concerning them, with regard to the criteria for their selection, their actual identification, and the training, supervision and follow-up that they are afforded. There are no magic bullets in dealing with the different issues concerning facilitators. Rather, it is important to consider holistically the approach to their recruitment, training, support and conditions of service.
Where Reflect is under consideration as an approach to be up-scaled beyond NGO projects, the cost of materials, training, refresher courses and supervision (depending on the degree of integration with existing Ministry programmes), is an issue in terms of the sustainability of the programme.
In several of the evaluations an overriding concern is expressed about the need to foster a 'culture' of evaluation within the programmes. However, the expected outcomes of many Reflect programmes have remained fuzzy, certainly not sufficiently well defined for indicator development. Similarly, output or outcomes of several programmes remain unassessed, making it difficult to develop feedback loops related to positive behaviours. The purpose of participatory evaluation where it has been used, has often been left vague, and in other cases, hasn't been invoked.
Other evaluation issues which emerged from the report concerned the fact that for any outcomes where there are diverse groups whose needs are to be met, it is important to distinguish adequately between these groups. Unless it is possible to hold constant certain factors (because one has collected data on them), it may be that the variation being explained is spurious or the attribution to particular factors is erroneous. Evaluations intended to inform potential up-scaling need to pay especial attention to such data gathering. On the other hand, the monitoring of a programme for an implementing organisation with a limited scope may not require such data collection. This underlines the importance of clarifying the purpose of an evaluation and whom it is meant to serve and matching the data collection with the purpose.
Most of the measurements of empowerment are relatively impressionistic, based on PRA and not distinguishing between different groups of participants, nor attributing, necessarily, action points to Reflect circles. Such measurements may be inadequate for bringing about the desired linkages between those traditionally supporting literacy and those supporting wider community development initiatives. For instance, were Reflect to be used to involve marginalized communities in contributing to poverty reduction strategies, the rigour of the change measurement possible from such an approach may be inadequate.
All of the evaluations reviewed emphasised the importance of participatory evaluation. Yet, the purposes and ways of involving participants varied considerably across the different cases. In some, the appropriateness of a participatory evaluation was merely stated as being obvious. No one would deny the importance of participants being involved in the evaluation of their own programmes. However, as the frustration of the evaluators expressed in several cases, the time required for participants to develop their own indicators and to feel that they truly had an involvement in the evaluation was greater than the time allowed. The preparatory work underlying successful participatory evaluation needs to start long before the evaluation begins. What we saw in several of the evaluations was that often there was confusion over the expected outcomes of the programme and the relative priority given to literacy and empowerment. In the absence of any prior baseline measures, the task was ill defined at best. Given the lack of clear, well-defined indicators and measures, it may not always be effective to design Reflect evaluations as participatory.
Recommendations
Three main recommendations are proposed in the report:1. Reflect should be applied in a cross-disciplinary way, across different types of development programmes, not necessarily even with an education bias, be they government or NGO-provided. One of the ways that this might be brought about is to emphasise 'voice'. Poverty reduction strategies require the voices of the poor to be heard so that strategies can be tailored not only at the macro level, but also to the needs of the poor at the micro level. This is understood, but the means by which such voices are heard are not institutionalised, and like any strategy that seeks to revolutionise practice, old habits die hard. Even if voices have been heard genuinely, without some community-based development organisation, it will be difficult to sustain the communication. Reflect, as a flexible approach that can be geared to a variety of purposes, could play a strong role in the development of such community potential.
2. Reflect programmes must develop a culture of evaluation amongst participants. The lessons of school improvement work bear scrutiny by Reflect practitioners. Whole school self-evaluation, for example, could easily be adapted to Reflect circles and their communities. It would entail assessment of the current status of the circle / community followed by the identification of priorities in a development plan. A shorter-term action plan could also be implemented and later evaluated, leading back to a new round of assessment.
If such regular self-evaluation were carried out, it would complement any external evaluations whose timelines tend to mediate against extensive involvement of this kind. Clearly there is a place for external evaluations, which may well be participatory in nature, but whose terms of reference serve implementing or supporting organisations or the government. However, the desire to make them participatory should not obviate the definition and measurement of outcomes. Such measures are as much a means of charting progress and modifying plans within communities as they are of providing benchmark tests for those supporting such communities.
3. Programme pointers
A set of pointers emerges from the evaluations for improving Reflect programmes. These are:
· To nurture the creativity and commitment of the facilitators (which involves seeing to their incentives and materials)
· To ensure the system of supervision works, especially if the programme is up-scaled to national level
· To integrate Reflect with community development - don't isolate it in education
· To provide regular refresher courses that meet the facilitators' needs
· To provide other courses of importance
· To organise meetings and exchanges between facilitators
· To resolve the literacy vs. empowerment issue in-situ
· To define outcomes and develop measures and indicators of progress as well as documentation guidelines
· To provide for feedback loops from evaluations
· To facilitate the monitoring of evaluations
· To ensure inter- and intra-agency coordination and cooperation
· To pay special attention to language issues
· To provide guidelines for constructing a baseline survey as well as testing in literacy and numeracy
· To strive toward the integration of Reflect in PRSP designs and within larger programmes, i.e. not literacy aloneThe report concludes that Reflect has much more promise than has been demonstrated in the set of evaluations reviewed. It remains for the international NGOs working with Reflect, to consider the proposed ways forward and to assist implementing organisations not only to improve practice, but also to ensure that the synergies implied in the new definition of Reflect are realised in support of poverty alleviation.
As practitioners of Reflect, many of the members of CIRAC (the International Reflect Circle) had direct experience of at least one of the Evaluations reviewed in Abby Riddel's report. We have our own misgivings about these Evaluations, which in the majority of cases were unparticipatory and carried out over a short period of time by consultants with little knowledge of the context or experience of working with Reflect. For a further discussion of these points and for proposals on methodology for future evaluations of Reflect, please see the forthcoming CIRAC document on Monitoring and Evaluation, also produced by this team.
The Evaluation documents themselves are long and often unwieldy. We therefore welcome the work that Abby has carried out synthesising and analysing these documents and drawing out key lessons and recommendations in a succinct and clear report. Hers was not an enviable task. Whilst there are a number of key points where we differ with Abby's analysis we feel that she has raised some valuable questions and many of the recommendations she makes will be extremely useful to practitioners in their work with Reflect and when looking at how to carry out future evaluations of Reflect. In fact, often our disagreement was not so much with Abby's analysis of a particular point but with the Evaluation on which this analysis was based - not always an easy distinction. Our detailed response to Abby's report is below:
Literacy vs. empowerment
Abby comments that a fundamental issue, raised in a number of the evaluations, is the relative priority given to literacy or empowerment in Reflect. She talks of the "renewed" definition of Reflect , which "presents an evaluation challenge that goes considerably beyond testing the achievement of basic literacy and numeracy" (Abby Riddel p 48). Whilst agreeing that the current definition of Reflect goes further than that given in the 1996 Reflect Mother Manual, we would argue that, from the beginning, Reflect has gone beyond basic literacy and numeracy and that empowerment has always been central to the approach. This was made clear in the evaluation of the three pilot projects carried out in 1995 (see ODA paper 1996) and can even be seen in the origins of the name "Reflect" (Regenerated Freirean Literacy through Empowering Community Techniques).We also feel that Abby makes a rather artificial distinction between empowerment and literacy / numeracy, which are in fact closely linked to the empowerment process in Reflect. Each Reflect project is uniquely adapted to its particular context and the focus of the work may change from place to place but it is rarely a question of a choice between working with empowerment OR literacy, rather these elements are interwoven. The conception of literacy is shifting from a reductive traditional focus on technical ability / competency to a focus on its practical value or use.
Where we would agree with Abby is in the challenge this presents for evaluators. It is not possible to measure empowerment in a one-off visit to a community or in a quick test and this was one of the key problems of all 13 Evaluations, only two of which took place over a period longer than 6 weeks. In some cases (for example, the Uganda, Okech and Ghana, NFED evaluations) the explicit aim of the evaluation was to make comparisons with the traditional government literacy programmes, working with a very narrow definition of literacy and paying little attention to empowerment outcomes. We recognise that future evaluations of Reflect will need to face up to this challenge, finding effective ways to assess the less tangible outcomes of a project. This may well involve a very different style of evaluation, one that is more systematic, thorough and ongoing than any of the 13 reviewed in this report, and that involves all the key stakeholders in a truly participatory process.
Links with other community organisations:
Abby talks of the "often missing relationship between Reflect circles and other community development organisations" and states that, "For the ideals of the "renewed" definition of Reflect to be realised, there needs to be wider understanding of the development and empowerment aims and the integration of Reflect practice beyond those using the approach" (AR p. 49). Whilst accepting that this may have been the impression given in some of the Evaluation documents, we would question this generalisation. The recent International Reflect Survey, completed by over 120 organisations working with Reflect worldwide, indicated that the majority of Reflect circles are closely integrated with community development work and that many circles were actually formed from existing community groups. Unfortunately, this is far more often the case in the smaller organisations working with the approach than in large organisations such as ActionAid and Save the Children, which, until recently, have tended to keep Reflect compartmentalised under "education" and quite separate from their work in other sectors. Although slowly, we feel that this is now changing as these organisations come to recognise the relevance of Reflect as an approach in a much wider arena.
Action Points:
Another issue highlighted in Abby's report is the need to tackle the resource requirements for Action Points - "it is important that [Circles] are not left high and dry just at the point when they are mobilising themselves" (AR p. 50). This is an interesting issue but one that we would approach from a slightly different angle. If an implementing organisation engages itself in ensuring the resources for every Action Point highlighted by the community it may find itself trapped in very traditional service delivery mode. The focus of Reflect work (and of development work in general) is shifting to enabling people to demand a more effective use of existing resources or to leverage new resources as their basic right, engaging with institutions such as the local government or relevant ministry, for example. Rather than depending on NGO handouts, local people are learning to recognise and challenge existing power relations, realising that the power to make changes is in their own hands.The team was quickly able to come up with a number of examples of where such empowerment has taken place. In Ghana, for example, faced with the problem of a shortage of teachers for the local primary school, the community education committee presented a demand for a new teacher to the District Education Authority. The demand was accepted and a new teacher provided for the school. Similarly, in both Pakistan and Ghana, community members successfully gathered together to demonstrate for the provision of a water supply system and in Bangladesh, Circle members successfully negotiated with local officials to repair a broken road. Often, the community is able to make significant steps without any external help. When a community in Mozambique wanted to improve the birth attendance facilities offered, they approached the District Health Officer to support midwife training, but the community itself was able to construct a new delivery shelter.
The Evaluation documents that Abby looked at present little evidence of this type. We feel that this is symptomatic of the failure of many of the Evaluations to capture the true spirit of Reflect, instead focusing on quantitative indicators aimed at evaluating literacy and numeracy skills acquired in the group. Although other materials documenting such experiences do exist, we feel that there are many stories that have yet to be told and it is up to all of us in CIRAC to make sure that these are collected and shared.
Facilitator training:
Abby raises some important issues about the training and ongoing support of Reflect facilitators. As she says, facilitators are the crux of Reflect programmes and it is important to consider holistically the approach to their recruitment, training, support and conditions of service (AR p.51). A number of important issues were raised in the Evaluations regarding this point and the CIRAC members recognise it as a key issue for the future sustainability of the Reflect approach. As Abby points out, Reflect may be cheaper than some approaches as it dispenses with any text books, but the skills required of the facilitator can have extra resource implications in terms of the training and ongoing support that are necessary. In an increasing number of programmes, facilitators meet regularly to discuss problems and successes amongst themselves and provide mutual support, seeking outside help when necessary. This is a non-hierarchical and cost-effective way of building on the capacity of facilitators. For further thoughts on the training and support of facilitators please see the forthcoming CIRAC document on Training.Abby also refers to the need for external training in areas other than Reflect by government and other specialists (AR p. 50). Whilst not disagreeing with this suggestion we feel that it is important to make sure that such training is in line with the approach rather than in conflict with it. Some of the training courses offered by government and other organisations is run along very rigid, traditional lines, not at all in harmony with the participatory approach of Reflect. We would also add that many of the topics mentioned, for example, micro-planning, leadership training and advocacy training, are already included as part of the ongoing training of Reflect facilitators in many programmes. In Pakistan, for example, the facilitators meet regularly to discuss their work and give each other support and "experts" in particular fields are often invited to talk to the group in an informal setting.
Indicators for Evaluation
We agree with Abby that in many Reflect programmes the expected outcomes are not sufficiently well defined for indicator development and there is a lack of baseline material, which is crucial for any type of evaluation. This is an important area to work on. We feel that it is essential to develop a culture of continual internal evaluation within Circles, so that the participants themselves are in a position to identify their expectations of a programme and to judge the extent to which these are fulfilled over a period of time. Evaluation should be integral to the Reflect process.Abby criticizes the PRA-based measurements of empowerment as being "impressionistic". Whilst we recognise the need for continued improvement in evaluations and the tools used in the evaluations of Reflect programmes, we would argue that some extremely valuable information can be gathered using PRA tools. However, it is not always possible to provide a rigorous measurement to illustrate the changes that have come about in the life of a community or individual as a result of their participation in a Reflect circle. Whilst we are committed to the constant sharpening of indicators for the evaluation of Reflect, we also feel that it is useful to listen to people's often-subjective impressions of how life has changed for them. Abby's report tends to ignore much of the evidence of empowerment which is present in the Evaluations (India, for example), on the basis that it was not collected in a rigorous / scientific manner. Although the conclusiveness of the evidence in any one project may be open to question, the accumulation of evidence from multiple and very diverse sites can be a useful pointer and we feel it should not have been ignored.
Participatory Evaluation
We are strongly in favour of participatory evaluations. As Abby states, all of the evaluations reviewed emphasised the importance of participatory evaluation. However, in our opinion, none of them actually involved the participants in any meaningful way. As she makes clear in her report, "the time required for participants to develop their own indicators and to feel that they truly had an involvement in the evaluation was greater than that allowed. The preparatory work underlying successful participatory evaluation needs to start long before the evaluation begins in clarifying outcomes, measures, etc.". She adds, "Given the lack of clear well-defined indicators and measures, it may not always be effective to design Reflect evaluations as participatory" (AR p. 56). We would approach these points from another angle, arguing rather than seeing the problems of time and a lack of well-defined indicators as a barrier to future participatory evaluations, we must accept the challenge of devising thorough, ongoing evaluations with clear indicators to be developed and applied in a process involving all key stakeholders. As Abby states, "None of the evaluations was geared specifically to the needs of the participants". We would particularly support the use of bottom-up evaluations in which the Circle participants themselves take part in an ongoing process to monitor their progress against expectations / indicators that they have set and in which evaluation is an integral part of the learning and empowerment process.Reflect is continually evolving and so, as practitioners, our approaches to evaluation must also continually evolve. We must accept the challenges laid down in Abby's Review as well as new challenges as they emerge. Indeed steps have already been taken to explore new ways of evaluating Reflect work, for example, in the Creative Evaluations of Reflect, which were carried out recently in programmes in five different countries, resulting not in paper documents but in videos and audio recordings for example, often produced by the participants themselves. In addition, and as mentioned at the beginning of this document, the CIRAC Monitoring and Evaluation Team is currently working on a publication exploring different types of evaluation and looking at possible directions for the future evaluation of Reflect (forthcoming).
By CIRAC's Monitoring & Evaluation Team, May 2001
Tabular Summaries of the 13 Evaluations
Bangladesh
Programme Context
Pilot in 10 circles in 1994, consolidation through 9 partners in 1995, extending to 36 programmes by 1999
Reflect Approach
27,000 participants – all but 500 women, covered by 36 implementing organisations, supported by Reflect Coordinating Unit; 2-month preparatory period, 9-month circle; terminal evaluation; 12-month post-circle period; common Reflect manual with guided keywords and graphics
Evaluation Purpose/Timing
End of five-year programme evaluation; July-August 1999; required for new funding proposal to DFID; to enable AA to make informed decisions about mainstreaming the programme
Evaluation Methodology
6-person team of external and internal evaluators; one month; literature review, meetings and discussions with AA and partners, learners and facilitators, and field observations; Coordinating Unit categorised implementing organisations by quality and 1 good and weak circle chosen from each of three categories plus 4 new organisations
Literacy/ Numeracy
Reported that 60% of ‘graduates’ have at least minimum competencies (lowest amongst adults, highest amongst adolescents) but in 1997-8 tracer study, levels of minimum competencies in numeracy were reported as between 8-14%; estimate that 40% participants achieve sustainable literacy
Empowerment
Growth in assertiveness and confidence only for the learners belonging to the good/strong circles, not the weak ones
Other Outcomes
74% participants using their skills in helping their children in their education, writing letters, maintaining household accounts, etc.; positive but impressionistic views of changes effected in women/girl
Weaknesses
Focus in Reflect on the more easily addressed practical, rather than the longer term, strategic gender needs and interests; insufficient interface with other development programmes to see through action points to socio-economic benefits; mechanical translation of issues and action points into stereotypical activities or potential actions that cannot be carried out
Recommendations
Importance of critical analysis and engendering more discussion and debate on Reflect
Upgrading monitoring and evaluation
Develop standardised and quantitative testing of literacy achievement that integrates set competency levels with the terminal evaluation at 9 mos.
Need for greater flexibility and creativity of facilitators – through adequate training inputs and enabling supervisions
Bring to scale
Endorse need for comparative study of different literacy approaches
Programme Context
One of three original pilots in 1993; 22 organisations now in network with CIAZO as coordinator; roots in popular education, focus on empowerment, democracy and sustainable local development
Reflect Approach
Perception that Reflect is more useful for organisation and empowerment than for literacy; 10-12 day training plus limited technical support
Evaluation Purpose/Timing
To investigate the process and impact of Reflect and to come up with lessons for improving its future implementation
Evaluation Methodology
Four people two months, between September and November 1999; documentation, interviews, workshop and observation of activities
Literacy/ Numeracy
N/A
Empowerment
Women less shy, more self-esteem; increased participation in circles and communities (though no prior measure)
Other Outcomes
Environment receives much attention; Reflect has enabled CIAZO to become a mediator of organisations in the region; false expectations leading to drop-out when people don’t get traditional literacy lessons
Weaknesses
Problems in the use of Reflect techniques; distance between theoretical understanding and practice; high facilitator turnover; insufficient follow-up; insufficient training and follow up; insufficient power analysis; problems finding facilitators
Recommendations
Need to clarify aims and role in community development and literacy; need to strengthen and improve training; training required in management processes to lead to plans of action at local level based on power analysis of different actors, development of more democratic and responsible leadership styles and development of negotiation and conflict resolution skills; need to consider how to make use of existing capacity in communities; need to analyse and improve all the activities from a gender perspective; need to study successful actions and learn from them
Programme Context
ActionAid supported in 5 communities in isolated area of Ghana, through local organisation, BILFACU; 79 participants; begun in February 1998
Reflect Approach
Weekly circle meetings – mostly women, facilitators men; adaptation of Ugandan manual; 9 days’ initial training; monthly facilitators’ meetings; weekly supervision; materials provided by BILFACU; refresher courses every 3 mos.; 3-monthly reports; covered 10-11 units; no incentives but working tools provided
Evaluation Purpose/Timing
At one year; 3 day evaluation in March 1999 to gauge how they would like to proceed and how far they had got over the year
Evaluation Methodology
Participatory, involving team of BILFACU and ActionAid staff, facilitator, community leader and representatives of interested organisations; focus group discussions, semi-structured interviews and circles’ time lines
Literacy/ Numeracy
Participants asked to write a short letter – most could write only a few words; majority could do addition, subtraction and multiplication, not division
Empowerment
Action points carried out successfully as long as no need for external financing; lots of energy and commitment and many changes and activities reported in communities, though action points derived uniformly from original manual; beginnings of some changes in traditional men and women’s roles reported in individual communities
Other Outcomes
Men more integrated into process than usual
Weaknesses
Dependence on external funding, not sustainable, underestimated possible achievements in literacy (undervalued)
Recommendations
Scale up; train facilitators in micro-planning techniques; train Literacy Committees in advocacy; investigate further factors contributing to the poor reading writing and numeracy skills; adapt facilitators’ guide
Programme Context
In 1997 ActionAid supported Non-Formal Education Division (NFED) of Ministry of Ed. to pilot Reflect in two districts.
Reflect Approach
251 participants (44% women); circles 4x week; fortnightly supervision
Evaluation Purpose/Timing
To compare Reflect pilot with NFED Functional Skills and Literacy Project and consider partnerships to extend coverage
Evaluation
Methodology
Meetings with field staff, participants; review of learning and teaching materials, interviews with community leaders and participants, development projects visited
Literacy/ Numeracy
Unclear though some could write simple letters
Empowerment
Community engagement, though threatened by lack of external finance of action points
Other Outcomes
Improved community interaction
Weaknesses
Inadequate materials, including follow up readinig materials, inadequate training of trainers, lowered selection criteria for facilitators
Recommendations
Extend Reflect not as project, but as “integral part of development process.”
Programme Context
Catering for landless labourers and marginalised farmers; different programmes started at different times between 1996-7
Reflect Approach
Varied approaches among organisations: between daily and weekly meetings, and between 20 and 60 days facilitators’ training, for instance
Evaluation Purpose/Timing
Commissioned by DFID (India) to investigate the sustainability of Reflect, its potential for upscaling, and the possibility of the convergence of Reflect with the national literacy campaigns; 8-months between September 1998 and April 1999 involving 4 national researchers; first draft March 2000
Evaluation Methodology
Three cases selected in under-developed regions supported by three different organisations; two ‘good’ and two ‘problematic’ circles identified by organisation in each region; group discussions with AA and partner organisations, trainers, facilitators and participants; interviews with participants, facilitators and community members/leaders in selected circles/villages; secondary data studies
Literacy/ Numeracy
Literacy component neglected where Reflect adopted in all areas of development work; difficulty of relating inputs and outcomes specifically related to Reflect; 25% estimated as attaining sustainable literacy after 2 years’ circles; results of NGO programmes comparable to National Literacy Programme in terms of literacy, though wider outcomes for Reflect
Empowerment
Different types of community actions detailed; enumerated number of collective actions per circle by organisation; evaluation raises the question of the limits of empowering processes given the inequitable socio-economic, political and gendered opportunity
Other Outcomes
At least 50% of those enrolled chose not to acquire literacy; cost per learner per year estimated at Rs.1313, for Adithi (Rs.451) – differentials Reflected in facilitators’ salary differences Rs.400 vs 1,000 per month)
Weaknesses
Need for Reflect to confront ‘the issue of alphabets in their own right.’; MKP’s ambivalence towards literacy was a factor why literacy levels had not progressed adequately; tendency to over-emphasise aspaects of ‘spontaneity’ and flexibility in the name of participation at the cost of the crucial role of leadership and its reponsibility for planning of implementation; continuing ambivalence btween prioritisation given to or the balance between literacy and empowerment
Recommendations
Reflect should be assessed on more than literacy achievement; other recommendations relate specifically to upscaling
Programme Context
Pilot of four NGOs in 7 districts over 3 years
Reflect Approach
Implementation depends on government Community Development Assistants (CDAs); facilitators with at least eighth grade identified by communities; CDAs responsible tor training facilitators, monitoring their performance, the progress of literacy circles and implementation of action plans; 804 participants (79% women) meeting in circles averaging 21 each
Evaluation Purpose/Timing
Mid-term review of pilot after 1 year; 2 weeks in December 1999; purpose: to investigate the management of Reflect, the differences it had made in circles and communities and to make practical recommendations based on the lessons learned
Evaluation Methodology
Five people, 1 external consultant and 2 each from national and regional offices; 4/7 districts chosen, one from each of the NGOs and chosen to ensure geographical diversity; each NGO selected 2 circles (1 strong and 1 struggling) and a control group (an average NALP class); focus group discussion with NGO and government staff, Reflect and NALP facilitators and participants; discussions held with community leaders; participants’ notebooks viewed; and informal testing administered of written and numeracy skills
Literacy/ Numeracy
Two-thirds of participants literate and numerate; in some circles wide variation; in ‘struggling’ circles fewer than 1/3 could read and write simple words; (inconclusive comparisons with NALP participants as few turned up to discussions)
Empowerment
Reported strong feelings of empowerment
Other Outcomes
A third to a half drop-out from successful and struggling circles respecitvely; most successful committees, those with greatest amount of cooperation from village leaders
Weaknesses
Lack of team work; lack of NGO-government staff cooperation; inadequate leadership to follow up action points; insufficient training and resources; little strategic planning; little done with indicators to monitor progress
Recommendations
Challenges for each group (participants, facilitators, implementors and steering committee) written up, including such things as:
Timely honoraria and provision of adequate materials
More support, supervision and refresher training
Encouragement for facilitators to adapt materials and extend
Improve documentation and monitoring of impact and making it participatory
Programme Context
Save the Children (USA) supported in 11 villages; 328 participants (M 71%)
Reflect Approach
Circles 1-6x/wk of 1½ -4 hrs.; covered 2-7 (all) themes of facilitators’ guide
Evaluation Purpose/Timing
At 6 months to strengthen programme and to gauge population’s opinion
Evaluation
Methodology
1 week (July 1999); 3 teams; meetings with 192 participants of 11 village circles, village leaders, facilitators, written numeracy and literacy test
Literacy/Numeracy
86-95% performed at average or good levels in reading, writing and arithmetic (most already exposed to some literacy training)
Empowerment
Reinforcement of community development initiatives such as collective cotton farming (response to question)
Other Outcomes
85-90% satisfied with programme; ¼ participants don’t attend regularly
Weaknesses
inadequate training and supervision offered facilitators low qualifications of facilitators
insufficient teaching materials’
too broad a range of participantsRecommendations
strengthen women’s participation
implement flexible meeting schedule
reinforce facilitators’ training and train 2/village
review supervision system and implement refresher training
implement participatory evaluation; support circle-initiated activities
establish better communication between circles and community
institute more collaboration between different SCF programmes
AMBIQUE
Programme Context
ActionAid programme begun 1995 in 2 provinces (Maputo and Zambezia) implemented by district directors of education; agricultural districts affected by floods and droughts, lacking agricultural inputs
Reflect Approach
Training of facilitators (2 2-week session); circles held 2-4x/week for at most 2 hours/day; no salaries, but contributions from communities; manual with 14 units
Evaluation Purpose/Timing
At 1½ years – to review the direction, process and evolution of the programme; to provide means of improving it; to gauge support to the Ministry of Education in its efforts
Evaluation Methodology
June 1999 during school holidays; in Maputo province only – 2 districts (350 participants recorded); literature review, interviews with participants, facilitators, local and central state and NGO bodies; external evaluator + AA country staff
Literacy/ Numeracy
‘all participants agree that...they have learned to read and write’ (unsubstantiated)
Empowerment
Link with community initiatives unclear
Effectiveness, Efficiency
Cost of $28,000 per year (numbers remaining in circles not reported, nor achievement indicators)
Other Outcomes
More than half dropped out
Weaknesses
Facilitators with insufficient prior education, limited literacy in local language, rigid use of manual; lack of follow-up supervision by district educaton department
Lessons
Mismatched expectations: local language vs. Portuguese; traditional vs. Reflect learning approaches
Recommendations
facilitator’s manual and other teaching materials should be produced in the community according to local realities
identification of facilitators needs to be carried out with community participation
there needs to be some assessment/evaluation of the level of the candidates, backed up by appropriate training, especially in teaching adults
careful attention needs to be paid to language – to use both the local language predominantly and to combine with Portuguese when appropriate
Programme Context First piloted in 1995 in 11 circles; end of 1999 had encompassed 700 circles, 200 then operating through 30 organisations in 28 districts; ActionAid Nepal superseded by Education Network in training
Reflect Approach
Diluted quality with expansion
Evaluation Purpose/Timing
December 1999; 2 weeks; 6-person team of external and internal consultants; to assess the concept, process, practice and impact of Reflect vis a vis other existing approaches and practices of literacy and empowering activities in the development context of Nepal
Evaluation Methodology
20 circles in 7 districts operated by 7 organisations selected by external consultants bearing in mind geographical and ethnic diversity; circles observed and discussions held of draft report with participants, facilitators, trainers and progamme managers
Literacy/ Numeracy
Most circles did not cover letters and literacy achievement was unsatisfactory
Empowerment
People have become vocal and in some cases have acted to address local issues; follow up activities almost solely involving saving and credit
Other Outcomes
More effective when run with the most marginalised peoples; reaching deprived and remote communities; school-age attendance seen to, including of girls
Weaknesses
Self-assessment of the training of trainers seems to have been waylaid.
The programme has become compartmentalised, covering particular issues, rather than being adapted to different localities.
Insufficient care had been taken in the selection of facilitators, choosing relatively well-to-do persons who, rather than serving as a bridge to empowerment, were seen as part of the oppressors.
The need to link local with national issues has been lost without the cross-fertilisation that would take place in exchanges amongst facilitators.
There have been insufficient refresher courses.
A secret aura has been placed around the training that has taken place – a lack of transparency.
There are insufficient materials for neo-literates.
Emotionally detached (from Reflect) people have begun running the programmes, without sufficient personal commitment.
PRA has been used as an easy way out of hard statistical work
The practice of giving an orientation on Reflect before an institution takes it up has been abandoned.
Objectives have not been clarified concerning literacy and community empowerment.
Follow-on activities have not always been conceived as part of the programme, nor future courses.
Environmental issues have been addressed in very limited and fragmented ways.
The roles and responsibilities of circle management committees have been unclear.
There is no clear assessment of the output of Reflect.
Recommendations
That an orientation to Reflect be compulsory for organisations wishing to utilise it, as had previously been the case
That systematic and continuous follow-up is required in order to assess the impact of Reflect
That there be regular Reflection and soul searching involving critiques of all sorts of Reflect practice and processes at all levels and that an analytical workshop of the trainers together with key players is urgent
That training be overhauled with an agreed minimum content and possible methods of delivery and tools and that professional trainers should have a particular length of practice and demonstrated ability
That re-orientation be compulsory for all trainers once every two years in order for them to continue to be trainers
That the main business of Reflect needs to be clarified, the place of literacy precisely defined and what is central to Reflect in terms of concepts and methods
That annual documentation of the changes in concepts and practices of Reflect be done nationally and globally (Reflect is rather weak in obtaining and processing factual data,..and achievements are not properly recorded.)
That Reflect be appropriately named in different contexts and languages
That the Reflect support mechanisms be upgraded, possibly catered for regionally and with some diversification of skills.
Programme Context
Pilot project in 11 circles in Northern Province begun in May 1998 Reflect Approach Two workshops for facilitators in February and March 1998 on PRA and linking literacy with participatory development (3 weeks in total); 1 refresher workshop 217 participants (9 men), average age 58, assessment forms and records of achievement, 176 hours/ year maximum circle time, average 93. Evaluation Purpose/Timing To assess how successful the pilot had been in making progress toward the alleviation of poverty through the creation of a practical link between literacy and community development, the establishment and support of critically-aware community groups and the trial and enhancement of Reflect. Carried out June-July 1999.
Evaluation Methodology Meetings and focus group discussions with participants, facilitators and a representative of the local council ward and a village chief. Written submissions from project manager and field worker
Literacy/ Numeracy
Participants could write their names and count to varying degrees. Empowerment Gaining self-confidence, beginnings of ideas of small development projects Other Outcomes Arranged for eye tests, drainage ditches, requested agricultural extension Weaknesses Require funding for continuation, insufficient focus on participants’ rather than community needs; facilitators and field worker need support and back up Recommendations Additional facilitator training; more focus on participants themselves, not just on community; more flexible meeting schedules; encourage fund raising initiatives; funding needs to be sought for Reflect survival; generic format needed for field worker’s reports; maintain project manager; consolidate links with local structures, especially local government; expand Reflect
Programme Context
WUS (UK)-funded women’s education programme to test out adapted version of Reflect amongst very poor communities in area including relief camps for refugees
Reflect Approach
Trained 30-40 facilitators between 22-35 days and 400 women learners, following survey of needs, using manual developed with 14 units about locale; 5 meetings/week for 9 mos.=
Evaluation Purpose/Timing
Final evaluation at end of 3-year programme; January 1999
EvalationMethodology
1 external and 4 internal evaluators over 4 days, interviewed participants, 35 facilitators, state adult education staff, the community, and investigated the wider literacy environment
Literacy/ Numeracy
Three meanings of literacy derived from programme: the acquisition and improvement of skills; nourishment and empowerment; and useful knowledge
Empowerment
See above
Other Outcomes
Frustration with obstacles to starting practical, income-generating projects
Weaknesses
N/A
Recommendations
Establish a sustained programme of training for facilitators
More creative use and development of literacy materials
Explore connection between numeracy gains and planning and budgeting for projects
Identify potential support for income-generating projects
Programme Context
Pilot project begun by ActionAid in 1993, evaluated and led to Co-ordination Unit being established; estimated total enrollment 10,000; 5 agencies besides AA utilise approach Reflect Approach Not described in report, but presumably same as in Archer, Cottingham, 10 days’ initial facilitators’ training, fortnightly training thereafter, refresher courses every 6 mos., field worker visits Evaluation Purpose/Timing Commissioned by ActionAid to assess the impact of Reflect on the communities in relation to empowerment, acquisition, application and retention of literacy skills and how this influences their attitudes, values and practices – to recommend strategies for improvement and assess cost-effectiveness Evaluation Methodology October 1999; 3 weeks; 1 external and 1 local consultant assisted by AA staff; interviews of participants, AA and partner organisation staff, literature review, home visit observations; graphics to assess participants’ achievements; consultative review workshop; covered 6 field areas (4 AA) Literacy/ Numeracy
All programmes teaching people literacy skills and participants have learnt and in many cases practising skills relevant and beneficial to daily livelihood Empowerment Not as significant changes in attitude and behaviour as in literacy skills Other Outcomes Gap between what AA offers and what participants wish to attain in terms of empowerment, skills acquisition and attitude change; emphasis on reading and writing to dtriment of empowerment and numeracy Weaknesses Lack of integration within AA projects; too much emphasis on the tools; particular aspects (e.g. gender, HIV/Aids and food security) suffocate other community initiatives; inadequate power analysis Recommendations Indicators of progress need to be wored out and analysed to assess performance; guidelines necessary for required documentation; clear objectives for new programmes to include beneficiaries’ interests and expectations
Programme Context
Integrated Non-formal Basic Education pilot project launched in 1992 in 1 district of each of the 8 regions; emphasised functional aspects of literacy and designed to help people improve their living conditions; expanded into national Functional Adult Literacy programme under decentralised management; total 1999 enrollment: 140,000 learners; NGO coverage of between 20-30,000; combined total covers less than 4% of the 5m non-literate adults Reflect Approach Pilot project begun by ActionAid in 1993, evaluated and led to Co-ordination Unit being established; estimated total enrollment 10,000 Evaluation Purpose/Timing Commissioned by the government via the World Bank to compare and contrast resource requirements and the effectiveness of the government-run Functional Adult Literacy programme and Reflect. Evaluation Methodology External consultant plus 4 Makerere U. staff members; 1999; random sampling and testing of 793 ‘graduates’ in 6 of the 8 regions; interview of 100 graduates; control sample of 20 non-literates in each of the 8 districts; written test of comprehension, numeracy and writing administered to ‘graduates’ and control group of Grade 3-4 primary school pupils; interviews of instructors, facilitators and village leaders Literacy/ Numeracy
Nearly everyone in the sample could read and understand simple questions, but the average numeracy score was 70% and the average score for more complicated comprehension questions was just over 60%; mean score on writing test was just under 40%; once prior schooling controlled for, there was little difference between the graduates of the FAL and those of Reflect; those with 3 or more years of primary schooling perform better with Reflect, while those with little to no schooling perform better with FAL
Empowerment Majority content with reading, writing and arithmetic in general; other benefits - meeting practical and strategic needs: improved family health, participation in civic activities, increased self-confidence Other Outcomes Graduates performed better than Primary 3 and 4 pupils; assessment of retention of literacy and numeracy skills not possible; functional knowledge, attitudes and practices of graduates better than for non-literate control group; length of exposure to literacy classes makes no different to knowledge score and not much to attitude score, but significant different to practice score; graduates spend more time discussion education and school matters with their kids and more time looking at their homework; average cost of FAL: $4 per participant per year; Reflect: $9; the quality of implementation is the major explanation of the variations in the attainments of learners Weaknesses Doubtful if numerous income generating projects would be sustainable – lack of market and necessary infrastructure; activities in which learners engaged are still at the bare subsistence level – hardly the kind which will enable many of them to break out of poverty; lack of post-literacy materials; instructors’ lack of incentives threaten programme; monitoring and supervisions situation is dismal Recommendations Ministry of Gender, Labour and Social Development should initiate discussions on options for a framework of shared responsibility for a national pool of adult educators; FAL Coordinating Unit should investigate how best to enhance quality of learning materials; multiply literature in indigenous languages; emphasise information on upgrading common rural occupations to link literacy, education and ability to raise productivity
