STRENGTHENING SOCIAL FUNDS: HUMAN RESOURCES FOR PROGRAM EVALUATION AND SOCIAL MANAGEMENT

Mario Torres

Paper presented at the meeting on "Social Funds and Poverty Reduction: Making Social Funds work for Poor People" Department of Economic and Social Affairs 15-16 October, United Nations, New York

This paper discusses how to strengthen social funds (SFs) through the upgrading of human resources for program evaluation and social management.  The argument is that although appropriate institutional arrangements is a critical factor for strengthening SFs, it can not be reached in the absence of qualified human resources with a capacity to assess social interventions and manage the complexity of social systems. It is also argued that creation of this capacity may be reached on the base of current program experience existing in SFs; local resources as represented by NGOs, universities and research centers; inter-institutional collaboration at the regional and international levels; and a renovated interest of the donor community on evaluating the impact of development aid. 

The questions to be responded are: What factors condition the institutional performance and strength of SFs? How institutional performance is related to expertise for program evaluation and social management? What lessons can be learned from experiences aimed at meeting these needs? What can be done?

Factors Affecting Institutional Strengthening and Performance of SFs

In a very sketchy way, performance and institutional strength of social funds are affected by factors that include the following:

• The institutional-political position of SFs. The closer the position to the policy-maker, the more vulnerable the SF is to political pressures and biases in service delivery.  This is particularly relevant when the clientele has a weak capacity to participate in the monitoring of services and the assessment of results;
• The institutional-administrative position of SFs.  Although a position outside the traditional state structure of sectoral ministries may facilitate program delivery, it may at the same time make difficult coordination of SFs activities with programs of other public agencies like the ministries of education, health, labor, or housing.  Coordination becomes a very important issue in the extent that program implementation is more based on decentralization towards local governments and communities. Decentralization of SFs activities may increase program ownership and sustainability of results but it may also make reaching program quality and effective monitoring difficult to obtain
• The institutional-program position of SFs within the range social services provided by other state and private agencies. SFs have been more focused on small and short-term activities.  However, the range of social demands reaching SFs goes beyond standard infrastructure works and increasingly includes attention to varied social demands coming from social groups such as youth, the elderly, or the "new poor".  SFs will need to find their most appropriate niche in this evolving context.
• The personnel turnover due to political shifts or changes in labor market conditions for professionals and technicians.  High personnel turnover makes difficult the building of an institutional memory and accumulation of experience for program design, monitoring and evaluation.  SFs have been quite effective for project design and implementation but have been less effective regarding project monitoring, evaluation, and maintenance.
• The eventual fluctuations of SFs financial resources particularly at the time of economic crisis or natural disasters.  The more dependent are SFs on external resources, the more vulnerable they are to changes in the agenda of donor agencies.

Most recent analyses and evaluations of social programs - which include SFs activities - indicate that failure of these programs to reach expected results is not due to action of one factor alone.  Rather, failure (or success) is the result of a number of combined factors - including some of the above mentioned. For example, program failure may be due to a combination of a clientelist program orientation, a centralized public sector delivery of services, and a poor monitoring capacity.  

There is a need to identify in a particular situation those factors that lead to successful program implementation, attainment of expected impact, and getting sustainable results.  This practice, more common in other development areas such as trade, technological innovation, or communication, is not much present in the reality of social service delivery.

The acknowledgement of the complexity involved in program success or failure is a quite important element in the current discussion about why social services for the poor fail.  Simplistic "one-factor" formulae like reliance on market mechanisms, more public expending, more reliance on the non-public sector, or unilateral donor interventions, have not been successful.  Actually, they have been identified as "not-to-dos" in the absence of broader diagnoses of the political, social and cultural context. 

"In most settings there are few evaluations of new interventions, and so no effective innovation and improvement in the productivity of services. Evaluating innovative service arrangements-such as new forms of accountability-is rarer still. If systems don't build in ways of learning about how to do things better, it should be no surprise when they stagnate. Relying on research from other countries, while useful, is not enough. Finding out how a particular intervention works in each country setting is crucial, since history, politics, and institutions determine what works, what doesn't, and why. Once again, this need not be." (World Bank, 2003: 26)

However, the application of knowledge to conduct these very basic diagnoses, to take advantage of lessons learned elsewhere, and to conduct the assessment of social programs is not as simple as it may appear.  It requires skillful human resources, professional experience, an enabling institutional environment, and access and sharing of public information.  These resources are scarcer than expected.

Institutional Performance, Evaluation Expertise and Social Management Capacity

Institutional performance and strength of SFs depend on the capacity to deliver, reach target populations, attain impact, and obtain sustainable results.  Examination of results produced by social services indicates that performance of social services has rendered mixed results. One general conclusion is that population sectors other than the poor, have benefited most from these services.  Among the reasons for this outcome is the existence of dysfunctional services to clients, low technical quality of services given by providers, and services not responsive to needs as identified by policy makers. This kind of problems indicate that it is not a particular link that has failed but the way in which clients, providers and policymakers as a system of interrelated social actors is approached (World Bank, 2003)

Although simple in its conceptual content and practical application, a policy framework aimed at dealing with clients, providers and policy makers as a system, requires a variety of skills and sound professional experience.  The reasons are varied:  social actors have different interests and agendas, divergence of interests requires conflict resolution strategies, alliances need to be identified, and appropriate information should be generated and shared.  In order to respond to these demands, the limited number of social service evaluations indicates that program operators need to be able to manage:

• Information systems providing public data about program objectives, target populations, services provided, groups reached, products delivered, and results;
• An effective strategy for reaching key social actors in the public and private sectors, and particularly at the local level in order to obtain alliances that contribute or facilitate social service delivery and sustainable results;
  Institutional arrangements that minimize risks associated with too close relationships with policy makers, and / or too far relationships with other public sector agencies;
• Institutional policies for generating and retaining qualify human resources, and preserving institutional memory; and, specially,
• Program evaluation aimed at improving sustainability of results

A particular problem that SFs face is the sustainability of results. Many SF activities consist of projects such as construction of schoolrooms, sanitary posts, small water and sewage systems, and irrigation works. SFs support also short-term employment programs and small production projects. Impact evaluation of these activities is difficult due to the lack of parameters of comparison particularly in areas where SFs are the main source of social investment. Unfortunately, SFs do not count on enough resources for following-up purposes.  Under these circumstances, assessing sustainability is not possible. Perhaps this is the most serious challenge for SFs and social service provision in general.

"…social funds and self-help interventions continue to face serious challenges of sustainability. One challenge arises from the cultural and social context of communities and their capacity for collective action. It is not clear that the self-help approach can bene?t fractured, heterogeneous communities that have little capacity for collective action. Alternative methods of service delivery may suit such poor communities better. But there is little factual evidence on this because evaluations typically do not compare social funds and self-help projects with conventional service delivery mechanisms. Nor do they take into account the negative side effects. The sustainability of self-help projects can be in jeopardy if line ministries or local governments ignore them once they are completed. Unless communities can ensure continuing support for recurrent costs and staff, they may not be able to sustain their project". (World Bank, 2003: 208)
     
Sectors such as education, health or housing are better positioned than SFs to follow-up project results provide monitoring and maintenance services, and obtain sustainability. SFs usually are dealing with very heterogeneous populations, have scarce baseline demographic information, provide services hard to monitor, operate under time constraints, and many times are subjected to pro-clientelist political pressures (World Bank, 2003) These working conditions require to re-examine the role of skills, training requirements, and knowledge sharing mechanisms needed to upgrade the existing capacities for managing and assessing social services.

Lessons Learned about Upgrading Social Management and Evaluation Skills

Difficult working conditions have not prevented SFs to produce relevant experiences for a better delivery of social services.  Actually, some key lessons relevant to upgrading social management have been obtained from innovative practices implemented by SFs.  However, lessons learned are few.  One reason is that SF's did not have as an objective the upgrading of its own human resources.  Another reason is that SF's have not recorded in systematic way their learning experiences

Although a few, lessons learned are relevant. A first lesson is that SFs have been able to develop a capacity to apply innovative administrative procedures in order to respond rapidly to local demands.  SFs have been able to overcome cumbersome bureaucratic and respond to community demands for water and sanitation services, construction of schoolrooms, alleviation of impacts caused by natural disasters or war, or food assistance, or employment emergency programs, to mention some examples.  A combination of an appropriate institutional arrangement and administrative skills make possible something usually considered unfeasible. 

A second lesson is that SFs have developed expertise to empower local communities.  Using a variety of methods, from participatory consultation to joint preparation of project proposals, SFs have facilitated local communities to learn how to identify their needs, establish objectives of social organization, articulate demands, and execute projects.  In the context of the tradition of centralized state structures that characterize many developing countries, this is an accomplishment not much recognized and praised.
 
Another lesson is that SFs operation has been almost free of corruption. Not much has been heard about corruption problems around program implementation by SFs.  Given the problems of corruption that have plagued public administration in less developed countries, the performance of SFs, in this regard, has been remarkable.  It is true that more attention needs to be given to some procedures such as implementation of bids and selection of providers in order to obtain more transparency and accountability.  But, in general terms, SFs have demonstrated that honest management of public funds is possible, it can be participatory, and it may be a learning experience for poor communities. 

Unfortunately, the situation regarding lessons learned from program evaluation has not been equally fruitful.  In this aspect there is a "negative" lesson.  SFs need to develop a capacity for program evaluation in order to improve targeting, obtain impact and reach sustainability.  This is not an easy task.  On one hand, social program evaluation in general is difficult due to the existence of other simultaneous factors affecting social outcomes.  Controlled experimentation is no possible. On the other, information and evaluation tools feasible to use under the circumstances SFs working in, are not well know.  Besides there is scarcity of human, financial and institutional resources for program evaluation purposes.

What Can Be Done?

The circumstances under which SFs operate require an innovative approach to program evaluation and training strategies in order to improve sustainability of results. The problem is not really the lack of methods and tools.  Neither is the lack of experiences to learn from (Marsden and Oakley, 1990). There are lessons to be learned from the practice in other development fields such as environmental impact assessment, participatory urban development, or rapid rural appraisal. The problem appears to be a combination of insufficient knowledge about non-standard evaluation methods, insufficient institutionalization of program evaluation, and political pressures on service delivery.  As a result the practice of evaluation as a dimension of program management is scarce.  " Although rarely carried out, some programs have tried to incorporate evaluation components to learn about the program. Mexico's Progresa explicitly included randomization and evaluation in its design" (World Bank, 2003:26)

Innovating program evaluation

Standard evaluation methods for the assessment of social programs, as mandated by social sciences such as sociology, economics, or social demography, are difficult to apply by operators of SFs.  Econometric modeling, correlation analysis of census data, surveys using random sampling, and experimental design require advanced training, are costly, and render limited useful and timely results.  As a consequence in practice, application of these methods is scarce.  Besides, experimental or quasi-experimental methods are difficult to apply. 

Unfortunately, the academic prestige and dominance of a positivistic tradition in social sciences have made difficult the application of other methodological approaches. "A major difficulty in assessing the impacts of any public program is that bene?ciaries are rarely selected randomly. Indeed, most programs are purposely targeted to speci?c groups or regions. Isolating the impacts from the circumstances that led to participation is then tortuous." (World Bank, 2003:212) However, although randomization is a useful tool in scientific research, it cannot be taken as a necessary condition to generate useful knowledge for program monitoring and assessment. 

"While positivist, reductionistic methods are often appropriate to pure social science endeavors which content themselves with identifying relationships between phenomena, such methods are less than ideal (neither effective nor efficient) in applied social policy assessment research which has the mandate to identify causal relationships-- difficult enough for natural science-- and to provide complete enough information about all significant effects of a policy, or all policy-relevant causes of a social condition, that sound policy judgments can be made." (Boothroyd, 1998)   

New research approaches, less costly, able to produce timely and useful results for orienting programs -although not necessarily to evaluate impact - may be applied. There is no need to invent them but to give information about their existence, results, lessons learned, and conditions of application, and to provide training for their use.  All these methods are able to produce knowledge about the client-provider-policy maker system, so critical to understand, manage and orient program for improving program results. The following is a summary presentation of some of them with no intention to be complete or exhaustive.

Participatory action research.  This methodology is based in the participation of people subjects or beneficiaries of the programs in setting research questions, gathering information, and helping in interpreting results. It requires collaboration between program operators, community leaders, beneficiaries, local "experts", central government officials and other stakeholders.  Variations of participatory action research are rapid rural appraisal, and urban participatory planning.

Systems Analysis.  This approach is based on systems theory and has produced "hard" (mathematical) and "soft" version. The soft version "…seeks only to identify directional changes (+ or -) in variables.  In assessment terms, this means it seeks to indicate whether social conditions (e.g. levels of health, education, job satisfaction) will get better or worse.  It does this through mapping the relationships among key systems components.  In this way, it is similar to modeling, but soft systems analysis makes no attempt to quantify the degree to which change in one component affects another.  It is content with indicating direction, including feedback." (Boothroyd, 1998)

Gender Analysis.  The attention is put in the position, roles, benefits, and demands of women in a particular system.  It produces knowledge by focusing attention on how women get access to services, employment opportunities, justice and social participation.  It is oriented to build a bridge between the analysis of values and norms at the micro level with policies and programs.  Its eventual "gender" bias may be overcome but including as a key component in the explanation of program results, how the interrelationships between sexes condition program outcomes.

Social Capital Assessment.  This is an emerging approach that puts the attention on the network of social relations and value practices.  It requires to examine program results in relation to how the individual, the family and the community are strengthen or weaken in terms of capacity to cooperate, put together resources, and commit to common endeavors.

These methodological approaches make use of a variety of heuristic tools. Some are communication tools such as workshops, community meetings, working groups, advisory committees, or community councils.  Others are exploratory techniques like preparation of planning processes, identification of program / project risks and opportunities, or setting of objectives. Finally, there are a number of data collection and analysis techniques like systems maps, brainstorming and brainwriting techniques, production of matrices, drawing of conceptual trees, or use of priority setting tools.

Common to all these methodological approaches and tools is that they do not require the type of sophisticated quantitative knowledge demanded by standard evaluation methodologies.  Illiterate persons may use some variants. This facilitates participation, application and rapid obtaining of results. 

Innovating training modalities

Standard training modalities have proved to be of limited usefulness to persons working in the public sector when the objective is updating, re-training, or new learning.  Job demands, time constraints, lack of financial resources or accessibility preclude SFs personnel to attend formal courses at training centers.  In other cases, knowledge required is not provided by available courses.  This situation has encouraged implementing innovative training modalities based on the exchange of experiences and inter-institutional collaboration.

Some training lessons have been learned from the experience of the Social Network of Latin American and the Caribbean (Torres and Pilotti, 2002):

• One lesson is that first-hand learning about successful program experiences is possible through short-term exchanges among SFs personnel.  Stages for a 4-5 day period may permit to get acquainted with new social service delivery modalities.  Examination of this type of experience indicates that to ensure learning, training expertise should support it. However, SF's have not made much use of local university or NGOs training resources for producing pedagogical guides applicable to short-term stages.  Neither a systematic follow-up of trainees has been conducted in order to verify whether new knowledge was applied.    
• A second lesson is that workshops aimed at discussing specific topics, such as how to improve service delivery to youth and women, how to make service effective and efficient in a context of decentralization, financing of poverty alleviation programs, or employment and income generation strategies, have proven to be useful for knowledge sharing and updating.  A small number of experts from different SFs use to attend to this type of events.  For this reason, there is a need to complement this kind of activities with mechanisms for further internal dissemination of knowledge learned.  This may be achieved by means of seminars or preparation of reports upon completion of the seminar and return to place of origin.
• A third lesson is that formation of a community of professionals working on poverty alleviation programs, coming from countries with different development levels and varied cultural backgrounds, contribute to strengthen social management capacities.  However, this result is not reachable through e-mail communication or electronic conferences only. Organization of electronic discussion forums has proved to be unsuccessful in the absence of personal knowledge among participants and clear practical results from the activity.  
• A fourth lesson is the need to use most advance information and communication technologies (ICTs) in order to obtain and effective, efficient and rapid communication among experts, program operators, and directors.  The use of electronic conferencing, for example, should be more frequent given the high costs of short-term stages, international workshops, and regional conferences.  Distance education modalities may complement short-term stages and workshops.  ICTs may also facilitate to go beyond the limits of groups of experts, and reach all relevant SFs personnel.
• A fifth lesson is the need to avoid an endogenous approach by SFs to training activities. They should make an effort to reach at the local level academic institutions, research centers, civil society organizations, and NGOs, while at the global level, reach international organizations and other regional networks. 

References

Boothroyd, Peter
1998 Social Policy Assessment Research: The Establishment, The Underground. A State-of-the-Art Report. Centre for Human Settlements. School of Community and Regional Planning. University of British Columbia.

Marsden, David and Peter Oakley (ed.)
1990 Evaluating social Development Projects. Development Guidelines No. 5 OXFAM Publications

Torres, Mario and Francisco Pilotti
2002 Red Social de America Latina y el Caribe: Lecciones Aprendidas y Perspectivas de Cooperacion entre los Fondos de Inversion Social. Organizacion de los Estados Americanos: Washington. DC. http://www.sdpgroup.com/documents.html

World Bank
2003 World Development Report 2004: Making Services Work For Poor People. Washington, DC.