This follow-up brief recapitulates the evaluation of the Colombian Technology Foresight Programme (CTFP). The foresight brief no. 119 (“Evaluating Foresight – The Colombian Case”) summarised the methodological aspects and principal focus of the framework on which the evaluation of the second cycle (2005 – 2008) of the CTFP was based. The objective of the present follow-up brief is to look back and summarise the implications of the evaluation of the CTFP by drawing on the key findings of the evaluation summary report. Concretely, it focuses on (1) the appropriateness and adaptation of the evaluation framework, as well as the effects for the spread of a foresight culture in Colombia that have been induced or stimulated by the evaluation of the CTFP and (2) the institutional mechanisms in support of the social appropriation of the CTFP’s output and results as well as the dissemination of the foresight knowledge generated by the CTFP to policy, industry and society as a whole.
Evaluation to Improve the Capacity for Learning
The development of Colombia’s Technology Foresight Programme (CTFP) has long been a reference point in the Latin American region. The CTFP is the first national foresight programme in Latin America that has been evaluated so far. The principal idea of the CTFP has been building a platform to create, distribute and utilise foresight knowledge in Colombia. It was intended to introduce new foci and new types of foresight practices and interventions in support of the strategic re-orientation of programmes and (sub-)sectors.
The focal point of the evaluation carried out in 2008 under the leadership of the University of Manchester was to reshape the objectives and activities of the second cycle of the CTFP (2005 – 2008). The conceptual framework of the evaluation was geared towards analysing foresight as a process.
This follow-up brief describes the methodological framework of the evaluation and discusses the learning process involved as well as the question whether the evaluation improved the aptitude for learning.
Evaluation of the CTFP
In Colombia, the evolution of long-term thinking in foresight has been largely driven by the role of COLCIENCIAS (Colombian Office of Science and Technology) as a node institution capable of facilitating inter-institutional alliances between various centres of excellence, on the one hand, and mobilising resources and engaging key stakeholders into a dynamic and self-reinforcing foresight learning process, on the other. Part and parcel of this learning process has been the comprehensive evaluation of the second cycle of the CTFP, which was geared towards identifying and supporting strategic sectors during the period between 2005 and 2008. Commissioned by COLCIENCIAS, the overall aim of the evaluation was to increase the CTFP’s capacity to shape and inform policy processes and actors.
Methodological Approach and Phases of the Evaluation Process
The evaluation of the CTFP was based on a methodological framework composed of a mix of seven diverse methods and activities that have been listed and described in some detail in the original foresight brief: (1) documentary analysis, (2) logic chart and indicators, (3) surveys, (4) interviews, (5) case studies, (6) benchmarking and (7) evaluation forum.
The evaluation process was divided in four phases:
Phase 1: Scoping – This phase had the principal objective to understand the main rationale of the evaluation process in order to design a coherent research process. The evaluation report states, “In addition to the traditional objectives of a Foresight programme evaluation (i.e. assessment of the impacts of the programme and the projects; assessment of the cost-effectiveness of the programme, and evaluation of the way in which Foresight is run in Colombia), COLCIENCIAS and SECAB [Secretaria Ejecutiva del Convenio Andres Bello] were particularly interested in identifying lessons and recommendations for the improvement of Foresight and horizon-scanning activities in the country.”
Phase 2: Understanding – This phase of the evaluation process was based on the collection of tacit and codified knowledge about the CTFP. “Tacit knowledge was collected through individual and group interviews with key stakeholders in COLCIENCIAS, other sponsors (e.g. ministries) and main stakeholders […]. Codified knowledge involved the compilation of major codified products (e.g. interim and final reports, books, journal publications and other important documents, such as individual project budgets and description of the programme’s expenses).”
Phase 3: Evaluating – This phase was based on a benchmarking of the CTFP against practices in other countries. The main objective here was to learn from other international best practices in establishing a national foresight initiative. The lessons shared from other countries included the UK, Malta, Russia, Spain and Hungary.
Phase 4: Learning – This phase involved conducting further analysis and preparing the final evaluation report.
Lessons Learned
The focus of the evaluation was on assessing what effects the second cycle had on policy and programme development. In particular, the impact of the CTFP on the design and of research policies was to be assessed as well as the effects of the CTFP on the promotion of national skills and the establishment of a national foresight culture in Colombia. In this sense, the evaluation was an important step towards synthesising the lessons derived from the national foresight exercise in terms of impact on skills, enhancement of capacities for strategic decision-making processes and policy design.
Key Findings of the CTFP Evaluation
The final report summarised the key findings of the report as follows:
(1) Regarding the overall objectives, the evaluation report states that the “CTFP objectives have been appropriate and successfully achieved. The programme has contributed to the creation of development visions and strategies for moving towards a knowledge-based society” through horizon-scanning and the building of foresight capacities in key sectors.
(2) As regards the value for money, the evaluation concluded that the CTFP achieved “a paradigm shift” by greatly contributing to the creation of a shared vision for “the productive transformation of Colombia into a knowledge-economy”. Furthermore, the evaluation states that the CTFP has begun to pay off since a diverse set of stakeholders have already adopted the vision brought forward by the CTFP in formulating their long-term objectives.
(3) Regarding the organisational structure of the CTFP, the evaluation observed that the institutional anchoring of the technical and decision-making groups in COLCIENCIAS during the second cycle notably “increased the CTFP’s capacity to shape and inform policy processes and actors. However, these changes also made the programme appear to be more of a COLCIENCIAS instrument than a national programme.”
(4) With respect to the approaches and mix of methods, the evaluation highlighted that “one original and effective feature of CTFP has been the combination of thee conceptual and methodological approaches: Foresight, horizon scanning and productive chain.”
(5) Regarding implementation and aftercare of the second cycle, the evaluation stressed the need for an aftercare strategy in the Colombian foresight programme. If the key support institutions of the programme “were to consider implementing such a strategy, this would probably increase the ability of Foresight to inform policy and shape research priorities. At the same time, it would also allow sufficient time for new networks to exploit the momentum created and consolidate institutional alliances.”
(6) With regard to the CTFP’s contribution to the spread of a foresight culture in Colombia, the evaluation states: “Some stakeholders still see foresight as being exclusively expert-oriented.” Therefore, it was recommended “that the general public be encouraged to participate in projects and training courses. This would probably require alliances with the private and productive sectors, in order to increase the financial and implementation feasibility of large-scale courses and projects.”
(7) Concerning the presence and visibility of the CTFP, the evaluation states that “while CTFP stands up well alongside programmes conducted elsewhere, it has limited visibility in the international academic and professional literature.” A clear implication derived from this is “that all major reports should be […] made available on the internet, and more widely disseminated through, for example, conference presentations and articles in relevant publications.”
(8) With regard to the impacts related to science, technology and innovation (STI), the evaluation stated that out of a total of 24 projects and more than 30 capacity-building courses, “nine projects had positive impacts on public and private polices and strategies; six projects had positive impacts on the agendas of STI programmes and institutions; five projects had positive impacts on the consolidation of research groups; two projects had positive impacts on the consolidation of S&T capacities; and two projects had positive impacts on international projects.”
(9) Concerning policy recommendations and strategies, these have been highlighted as “fundamental elements of CTFP outputs”. According to the evaluation, “the most significant influence of the CTFP on public policy has been the work on the STI Vision 2019.” Moreover, the CTFP’s biotechnology project had a significant influence on the policies and research priorities of COLCIENCIAS’ National Biotechnology Programme.
Networking Key to Spreading Foresight Culture
The strengthening of networks was a central pillar for achieving the principal purpose of the CTFP. In this sense, the enlargement of inter-institutional networks was instrumental for the realisation of a strategy to spread a foresight culture in Colombia. On the side of the sponsors and organisers of the CTFP, building networks to facilitate the involvement of different expertises in support of an interdisciplinary approach to foresight in many fields and sectors was a vital aspect for all those who participated in the CTFP.
Dissemination Strategy Falls Short of Potential
The CTFP produced a large number of high-quality scientific outputs. However, the strategy to disseminate the preliminary and final results has pursued an approach not conducive to enhancing the international visibility of the CTFP as a best practice across the region. Moreover, in disseminating the new knowledge generated, there was a tendency to emphasise tangible over non-tangible outputs. Since the degree to which the newly produced knowledge contributes to the opening of new strategic options or the strategic re-orientation of sectors is an important success criteria for foresight, it is vital to embed this new knowledge in people’s and organisations’ practices. This requires that the dissemination of the results be tailored to different target groups, which also calls for different means of communication appropriate to the respective group. Although the CTFP delivered many tangible outputs, such as scientific publications (predominantly in Spanish), the approach chosen to disseminate the outputs and results remained far behind the possibilities of alternative approaches that could have enhanced the interaction between the different governmental and scientific communities or increased the international visibility of the CTFP as a best practice from which others could learn.
Methodological Progress Towards Context-sensitive Use of Methodology
The choice of the methods is the most distinctive feature of a foresight process. The evaluation of the CTFP stated that “an average CTFP study involved more than 10 methods, with more or less half of these being horizon-scanning techniques (including bibliometrics, trend extrapolation and patent analysis) and the other half related to Foresight and productive chain approaches (e.g. scenarios, brainstorming, stakeholders mapping, key technologies, morphological analysis, among others).” Regarding the use of foresight, the evaluation focused on the selection of methods but did not monitor how well the methods used lined up with the STI implementation environment, specifically in regard to strengthening capacities in support of policymaking in the area of science, technology and innovation with due regard to local concerns in Colombia. Therefore, a future monitoring or evaluation framework might also focus on adjusting or reconfiguring the methods applied to fit the foresight process needs in Colombia.
Towards Enhancing Colombia’s Foresight Capabilities
Given that there is no ‘one-size-fits-all’ approach to evaluate foresight and that the evaluation of foresight activities cannot be carried out independently of the national context (Georghiou and Keenan, 2006), it is difficult to assess the interaction between foresight and evaluation with respect to the impact on policy instruments or the improvement of overall system coordination through strengthening linkages between innovation actors.
The concepts of effectiveness, efficiency, appropriateness and behavioural additionality on which the framework to evaluate the CTFP rests are key to understanding the focus of foresight activities that are carried out to reconfigure institutional set-ups and re-orient policy goals. Georghiou and Keenan (2006) state that the “[…] evaluation of foresight must include understanding of the interaction of foresight outputs with the strategic behaviour of policy and economic actors.” In this sense, the evaluation of the second cycle of the CTFP was an important step towards better understanding the drivers of the strategic behaviour on part of the key implementing institutions in the Colombian system. However, further advancing Colombia’s foresight capacities depends to a high degree on the institutionalisation of foresight in the Colombian context.
Improve Dissemination Through Alternative Communication Channels
According to the results of the evaluation of the second CTFP cycle, a key lesson that can be drawn refers to the appropriation of the insights gained from the foresight programme. Although the fact that the foresight programme was conducted by COLCIENCIAS was an important institutional driver, the lack of an aftercare strategy constitutes a weak point for the strategic influence of the foresight knowledge generated on the target groups and sectors. Therefore, an important recommendation to increase the future impact of the CTFP is that alternative dissemination and communication channels should be exploited to a greater extent. An improved dissemination strategy should also take into account the need for a stronger diversification of foresight capacity building in Colombia.
Towards Further Institutionalisation of Foresight in Colombia
The “shift from networks and individual exercises [….] to more institutionalisation towards centres of excellence“ is an important step to “take on responsibility for preserving knowledge and for allowing lessons learned to be carried forward in a long-term framework“ (Popper et al., 2010). In this sense, the evaluation revealed that a move away from the somewhat centralistic approach to anchor the foresight process in COLCIENCIAS towards a more effective institutional mechanism was a necessary step to better embed foresight in the Colombian STI system. COLCIENCIAS recent decision to institutionalise the foresight practices in the framework of the CTFP by establishing the Colombian Foresight Institute (COFI) at the Universidad del Valle (Cali) can be considered an important move to enhance the aptitude for learning and thus strengthen the contribution of foresight to reorienting the Colombian STI system. In this arrangement, multiple organisations will be able to conduct foresight.
Under the bottom line, we can conclude that the evaluation came at the appropriate time to develop recommendations on how the foresight outputs, results and knowledge generated during the second cycle of the CTFP could be better appropriated by the stakeholders and embedded in a broader strategic policy context. In particular, an improved dissemination strategy and the search for alternative ways of institutionalising foresight are central pillars for engaging future resources and a broad set of stakeholders in a dynamic and self-reinforcing learning process based on which a foresight culture can develop in line with the evolving STI policy system in the Colombian context.
| Authors: | Dirk Johann dirk.johann.fl@ait.ac.at | |||||||
| Sponsors: | COLCIENCIAS (Colombian Office of Science and Technology) | |||||||
| Type: | Evaluation of Foresight Programme | |||||||
| Geographic coverage: | Colombia | |||||||
| Organizer: | COLCIENCIAS (Colombian Office of Science and Technology)
PREST / Manchester Institute of Innovation Research (MIoIR), MBS, University of Manchester |
|||||||
| Duration: | 6–9 months | Budget: | € 40k | Time Horizon: | 2020 | Date of Brief: | May 2012 | |
Download EFP Brief No 218_ Embedding Foresight in the Colombian Innovation System
Sources and References
Georghiou, L. and M. Keenan (2006), “Evaluation of National Foresight Activities: Assessing Rationale, Process and Impact”, Technological Forecasting and Social Change, 73, 761-777
Popper, R., L. Georghiou, M. Keenan, I. Miles et al. (2010), Evaluating Foresight – Fully-fledged Evaluation of the Colombian Technology Foresight Programme (CTFP), Colombia: Universidad del Valle