Applying multicriteria analysis in on-going evaluation of EU structural programmes

Abstract

Keywords: Criteria for decision making; Planning policy; Regional policy

Introduction

In general, evaluation aims to achieve an appropriate level of information and knowledge so that the policy makers can adopt decisions regarding single projects or families of projects proposed in specific programmes (Owen, 2006) in a more appropriate way. The type of information and the level of detail vary depending on: i) the type of evaluation (ex-ante, in itinere or ex-post); ii) the characteristic of the investments; iii) the number and dimension of the projects included in the programmes; iv) the technical structure of the programmes. The information, gathered and elaborated by the evaluation process and offered to the stakeholders of the programme (or project), highlights what in the programme (or the project) ∗

This paper has been financed by the public fund PRIN - Project of Relevant National Interest ”Cost of lack of prevention in Italy. A research project aimed at the safe of the population and the proper distribution of public resources”, national coordinator prof. Alberto Prestininzi, project unit of University of Molise, Campobasso, local coordinator prof. Claudio Lupi. † E-mail: massimo.bagarani@esaricerche.it, Via Plinio 44, 00193 Rome, ITALY. +393204309108

works and what does not work as expected ((Berk and Rossi, 1999);(Chen, 2004); (Chen, 2005)), in terms of programme’s performance but also of priorities achieved (Rossi, Freeman, and Lipsey, 1999). Based on the results, the stakeholders can adopt the appropriate choices for solving problems, overcoming bottlenecks and managing a better designed programme ((Weiss, 1972); (Cronbach, 1982)). Chen in his works proposes a distinction between ”change” model and ”action” model arguing about the theoretical basis of programme evaluation (Chen, 1990). Starting from this theoretical framework, in this work we will utilize a specific multicriteria evaluation technique, the ELECTRE TRI ((Roy and Bouyssou, 1993); (Roy and Mousseau, 1996)). Our main scope is to demonstrate that this approach could represent an innovative and effective tool in processing the in itinere evaluation of programmes during the implementation phase, through the identification of Measures which present several difficulties in their implementation. It could therefore represent a valuable tool for decision support for the stakeholders of the programs. In this work, the model TRI was applied to the Regional Operational Program (ROP), operated by a southern Italian Region (the test Region) during the period 2000-2006, within the mid-term evaluation framework. In the application, we collected all information, results and considerations emerged during the evaluation through interviews, meetings, and analyses performed by the evaluator in respect of key stakeholders and beneficiaries of the implementation. The work is structured as follows: the typical structure of a Communitarian operational program is described in section 2 and a brief discussion of the role played by this type of intervention in the policy choices of the structural development of EU and Member States is provided; section 3 is devoted to presenting the ELECTRE TRI method adopted for the evaluation of the selected program; in section 4 we proceed to an application of the method to the case study represented by the operational program of the test Region; in Section 5 some concluding remarks are provided about the systematic application of the proposed method.

Structure and Logic of the Regional Operational Programs (ROP)

From a historical point of view, the political approach utilizing ROP started with the first cycle of Communitarian planning, in the period 1989-1993, inheriting the experience gained in previous years with the Integrated Mediterranean Projects (IMPs). The choice of the former European Economic Community (EEC) was twofold: i) to intervene with economic policy actions to address the problem of structural differences between areas of the member countries, ii) to assure that the Regions, and no more the States, are the centre of the Communitarian action in terms of territorial jurisdictions. This choice implicitly rejects the neoclassical hypothesis of ”natural” convergence of the economic systems, while assuming that public spending has a decisive role in overcoming the structural imbalances. Through the preparation and implementation of structural programs at regional level, public spending has the main purpose to achieve the priorities of the macroeconomic structural development: the economic and social convergence, and the preservation and the improvement of the environment ((European Economic Community, 1987); (European Union, 1992); (Monti, 1996); (The European Commission, 2004)). In the cohesion policy, Re-

gions have the responsibility to implement and govern ”from below” a public system structured on several levels of jurisdiction, the so-called multi-level governance, ((Scharpf, 1994); (Marks, Hooghe, and Blank, 1996); (Mayntz, 1999); (Liesbet and Marks, 2001); (Boccia and et al, 2003)). In this context, the operational program represents the most important planning tool, and it is the actual way to operate with the public expenditure in the regional territories. Figure (1) shows the general structure of a Communitarian program divided into Axes, Measures and individual projects within the Measures. The arrows identify logical causal connections among projects, Measures, Axes and Program. In summary, the set of projects within a Measure contributes to the achievement of the objectives of the Measure. The realization of a set of Measures allows the Program to achieve the priorities at Axis level and eventually the realization of the Axes allows the Region to obtain the priorities of the Program.

Figure 1: Scheme of the Structure of a Communitarian Program Because of the interdependence between each level of the hierarchy, some projects or Measures will contribute to meet both the specific objectives of the Measure or the Axis of belonging and the objectives of other Measures or Axes (functional relationships represented by the dashed lines in Figure (1)). During the 2000-2006 programming period, the Measure represented the ”atomistic” unit in the ”program cycle” (again, within 2014-2020 programming period, the Measure plays the same role). This implies that all government actions (monitoring, evaluation and reprogramming) were carried out primarily on the basis of the results obtained by the Measures during the program execution. Moreover, the definition of Measure provided by the Council Regulation 1260/99 in article 9 (j) stated that it was the means by which a priority is implemented over several years which enable operations to be financed. In general, the Measure plays a key role in controlling the program’s performance in different

evaluation phases: before (ex-ante), on-going (in itiner e) or mid-term, and after (ex-post). In particular, evaluators have a deep interest in the expected or actually achieved results from projects during the ex-ante and ex-post evaluation phases; on the other hand, the Measures become more interesting during the in itinere evaluation, precisely because they represent the centre of administration and program management. It is always possible to realize at least two types of analysis, in an interim evaluation at level of Measure: i) an evaluation of the effects generated by the complete realization of one or more Measures (Brancati and P., 2003); ii) an evaluation of the program implementation under a technical, political and administrative point of view. In the first case, the evaluation will produce an estimate of the expected effects, either for individual projects, or for Measures, or for the entire Program. In the second case, the analysis is performed on the entire set of Measures, assessing their performance, based on physical, financial, economic, and procedural indicators. In the latter case, it is therefore possible to verify the applicability of a methodological approach that, starting from an accurate assessment of the critical issues related to the progress of the Measures, allows the management bodies of the program to improve their decisions.

ELECTRE is a multi-criteria method based on outranking relations used to select, sort, or classify a set of alternatives with respect to multiple criteria. In particular, the version of ELECTRE TRI allows a classification of alternatives with respect to some equivalence classes predefined by the user. From a formal point of view, in the multi-criteria methods, a set of alternatives, or shares, A of variable size (coinciding with the number of Measures of a Regional Operational Programme, in the present case), can be analysed using a multi-criteria decision aid tool. With reference to the literature ((Roy and Bouyssou, 1993), (Roy and Mousseau, 1996); (Mousseau and Slowinski, 1998); (Lourenço and Costa, 2004)), we have three different ways to analyse the actions (in this case Measures): selection, classification and sorting. In the case of selection, a sub-set A1 ⊂ A of accepted actions and a sub-set A − A1 of rejected actions are defined, minimizing the number of elements contained in A1 . In the case of classification, the elements of A are assigned to a predefined set of categories based on the value of each alternative a, without any pairwise comparison with other alternatives contained in set A. Finally, in the case of sorting, a system of preferences on the set of alternatives that belong to A is defined. Given the political and economic characteristics of the elements analysed in this paper (the Measures of a ROP), it appeared advisable to choose the method of classification, excluding both the hypothesis of rejection, hardly acceptable in the context of the program implementation, and the hypothesis of sorting, which would limit too much the choices of the policymakers. ELECTRE TRI is a well-established method of classification characterized by allocation of alternatives to predefined categories1 . The assignment to a given category occurs through the identification of a number m of non-default criteria gj , with j ∈ F being F = {1, 2, ..., m} the set of indices of g, and the comparison with a number p of profiles bh , with h ∈ B being B = {1, 2, ..., p} the set of indices of b. The definition of the p profiles generates a set of p + 1 1

In this work the method has been entirely reconstructed within an Excel spreadsheet and subsequently applied to the Measures of the Operational Programme of the test Region.

categories such that bh constitutes the upper limit of category Ch and simultaneously the lower limit of category Ch+1 . Schematically, this approach is represented in Figure 2.

Figure 2: Representation of the Distribution of Criteria, Profiles and Categorization in the Case dell’ELECTRE TRI Source: representation derived from Mousseau, Slowinski, 1998. If we indicate with gj (a) the value assumed by the actions in the j−th criterion and with gj (bh ) the corresponding value assumed by the h−th profile, then the relationship of preferences O (outranking) will be such that aObh when the assertion that ”a is at least as preferable as bh ” or, in other words, that gj (a) ≥ gj (bh ), is verified2 The classification method also considers the introduction of inter-criteria parameters consisting in: indifference thresholds qj , preference thresholds pj , veto threshold vj , and the weights wj . Therefore, this leads to construct a relation O of preferences articulated in at least three preferential orders: indifference (I), weak preference (Q), strong preference (P ), in general assuming that I ≺ Q ≺ P . In the present case:   if aIbh if aQbh   if aP bh

then gj (bh ) − qj (bh ) < gj (a) ≤ gj (bh ) + qj (bh ) then gj (bh ) − pj (bh ) < gj (a) ≤ gj (bh ) + qj (bh ) then gj (bh ) + pj (bh ) ≤ gj (a)

A residual fourth category (R) ”corresponds to the absence of clear and positive reasons that 2

Without any loss of generality, we can assume to be in the presence of increasing preferences with increasing values of the criteria. Hypotheses of decreasing preferences with increasing values of the criterion will also be used in the subsequent analyses.

may justify any of the three previous reports” (Roy and Mousseau, 1996) and, consequently, represents a condition of lack of comparability. Based on these four preferential relations, the following sets of variables, of indices and of procedures were built: i) partial indices of concordance cj (a, bh ) and cj (bh , a); ii) partial indices of discordance dj (a, bh ) and dj (bh , a); iii) general index of concordance c(a, bh ) and c(bh , a); iv) fuzzy variable on the degree of credibility of the preference relation σ(a, bh ); v) derived variable of binary relations of preference (crisp relation or ”outranking” relationship); vi) procedures for allocation in pessimistic and optimistic hypothesis. The application of the method requires particular attention, especially for what concerns the development of the phases relating to: • definition of the criteria gj ; • definition of the profiles bh ; • definition of the values assigned to the weights wj . The definition of the criteria should comply with the principles underlying the valuation processes as well as briefly described in the introduction. In particular, in this case, the choice should be made in consultation with stakeholders, based on the proposal put forward by the evaluator, taking into account the characteristics of the implementation process and the findings of several investigations to officials responsible for the implementation of the Measures. Likewise, the definition of the profiles should also take into account the priorities of the decision maker (DM) when assigning values to the criteria gj for each profile. In this regard, the assessment involves the use of optimization techniques that, starting from what is expressed by the DM, define the optimal values of gj(bh) and the thresholds of indifference and preference, in order to minimize the differences between the values assigned by the ELECTRE and the assignment made by the DM ((Mousseau and Slowinski, 1998), but also (Rogers and Bruen, 1998); (Dias and Clı́maco, 2000)). In this work, the assignment was made by the evaluator, which assumed the role of DM, in the light of the available information. Although the optimization method has not been applied, the differential variables (slack variables) of Mousseau and Slowinski, however, have been calculated. Finally, this paper adopts the hypothesis of absence of differences in the related importance of criteria (the weights), assuming wj = 1 for all criteria ((Roy and Mousseau, 1996); (Figueira and Roy, 2002)).

Application to a 2000-2006 Regional Operational Program (ROP)

The 2000-2006 Community Support Framework (CSF) for Italy, following the provisions of the Community regulations, in particular the 1260/99, provided for the formulation of ROPs, based on the scheme proposed earlier in figure 1 ((Ministero del Tesoro, 1999); (Ministero del Tesoro, 2000); (Unione Europea, 2000)), by the Objective 1 Regions and by Regions under the transitional support. The CSF provided an articulation of programs in six axes: 1. Natural Resources; 2. Cultural resources; 3. Human resources; 4. Local development systems; 5. Cities; 6. Networks and nodes. The definition of Measures, actions and projects is Region’s competence. The test Region chose to develop its own Program in just five Axes, giving up the Axis City, and in 43 Measures. During the implementation, the Program has been characterized by an evaluation activity, carried out by an independent evaluator, and by technical assistance to support the choices of the Managing Authority (MA). In the first phase of implementation, the attention of these three players, evaluators, technical assistance and MA, was mainly directed to identify the existence of any difficulties in realization within the 43 Measures, with particular reference to bureaucratic constraints and administrative delays, or inefficiencies, due to the complexity of the management of the Program. This evaluation activity was carried out through a set of actions: a) direct inquiries at the offices of the heads of individual Measures; b) repeated meetings with the officials in charge of the Measures; c) submission of questionnaires to representative subjects of economic and social partnership; d) submission of questionnaires to the beneficiary of actions envisaged in the Program; e) systematic verification of the results obtained between the three actors: evaluators, technical assistance and MA.

Based on the findings of the above activities, the study focused on four indicators of implementation (four criteria, so that m = 4):

4. Numbers of changes in the position of MM occurred during the time laps.

Number of Managers for each Measure (MM) - Criterion 1 The indicator varies from a minimum of zero (absence of MM) to a maximum of 1. The case with a value equal to unity is the most frequent with 31 cases out of 43. The second value in terms of frequency is 0.11 which is present in 9 Measures. The value of 0.5 is present in only 2 Measures and zero in 1 Measure. The value of 0.11 is determined by the fact that the same manager was responsible for nine Measures of Axis 3 ”Human Resources”, as well as, in the case of 0.5, the same manager was responsible at the same time for two Measures, always included in Axis 3 ”Human Resources”. The absence of MM (zero value) is relative to only Measure 4.15 ”Natural disasters” not yet activated at the time of mid-term evaluation. In order to achieve maximum administrative efficiency and the maximum social and economic impact, the decision maker should aim at minimizing the number of cases in which an MM is only partially involved in the implementation of the Measure. In other words, an efficient management should appoint one responsible manager for each Measure in order to ensure the best possible partnership in each action. A direct relationship between program efficiency and number of MMs is expected. Number of officials assigned to support the MM - Criterion 2 The structural complexity of the Communitarian programs requires an administrative organization where officials able to help to implement the program in the faster and the more efficient way support the MMs. In many cases, in the course of investigations, MMs complained about the shortage or lack of this support and the consequent inability to fully discharge by the necessary speed, the obligations imposed by Community rules. Also, the existence of a direct relationship between this indicator and the utility function of the decision maker is assumed, while stressing the existence of at least two caveats: i) the number of officials in support of a Measure will also depend on the importance that the DM assigns to the Measure in the broader context of programming and the economic and social development of the Region; ii) the growing of the number of officials support do not necessarily lead to an improvement in the efficiency and effectiveness of the management of the Measure, rather an excessive number of units, could drive to diseconomies in the implementation of the Measure. Regarding the first point, it is difficult to imagine an explicit DM statement on the importance of each Measure in terms of development policy for the Region. Actually, an indirect indication could be inferred from the financial values associated to each Measure in the program, even if the indication could be partially distorted by the decision of policy makers to simultaneously activate other policies financed by funds from other sources. A simple analysis of the correlation between the two variables (number of officials and financial dimension), for the 43 available cases, shows the substantial absence of a positive relationship. Regarding the second point, the analyst should verify the optimal number of officials to engage in the realization of the Measures, without any prejudice on the relationship between the number of officials and speed and quality of the execution of the program. Percentage of payments compared to the total of the amounts available - Criterion 3 This is an indicator usually adopted in the analysis of public expenditure and allows a measurement of the financial progress of each Measure. Also in this case, several factors may affect

the value of the share: i) a greater procedural simplicity; ii) the existence of specific structural characteristics that may affect the flow of spending; iii) a greater efficiency in management practices; iv) a more effective demand response; v) a better interpretation of the actual needs of the area; vi) a more adequate preparation of the communication and dissemination activities. Therefore, conditions of greater preference are reasonably associated with higher values of the ratio payments/programmed by the program managers, as in the two previous cases. The distribution of the indicator is bimodal where the first modal class equals to zero (no payments), which, in economic terms, means the failure to initiate Measures that fall into that class. This is not justified by procedural or market-related reasons, given that, in the tested Region, the Program started four years before. Actually, the average financial Program does not seem to have been proceeded at very high speed, at the date of December 2003, but this is a quite common condition in multiannual Community programs rather articulated in the destinations of expenditure and rather complex in procedural execution. On the other hand, a too low level of budget execution should generate an alarm in management, given the limited time frame to complete the payment of the programmed amounts. Numbers of changes in the position of MM - Criterion 4 This indicator indicates the number of changes of managers occurred, in each Measure, during the Program until the expiration date of the reprogramming (31 December 2003). Decreasing preferences are assumed in this case. The succession of managers inevitably generates delays, although in some cases it may be essential and also efficient. However, when the shifts become too frequent, the implementation of the Measure is seriously affected both in terms of efficiency and effectiveness. In fact, the new responsible manager, not only tends to partially modify some administrative aspects related to the implementation of the planned operations, but also needs to restart the contacts with the partnership and with beneficiaries and to solve the structural problems that limited the implementation of the Measure, that were, most likely, behind the changes themselves. The indicator varies from a minimum of zero to a maximum of five changes; this is a very high value considering that the actual implementation period covers a time span of just three years. Only five Measures showed no change in the management during the three years of effective implementation, while 15 Measures experienced one change and 29 one or two changes. A rather large group of 9 Measures showed 5 changes. The correlation matrix (see Table 1) shows the absence of redundancy among the four indicators, with the exception of the high correlation value between ”Number of Managers for Measure (MM)” (Criterion 1) and ”Number of officials assigned to support the MM” (Criterion 2). Table 1: Correlation matrix of the indicators chosen as criteria Criterion 1 Criterion 2 Criterion 3 Criterion 4 Criterion 1 Criterion 2 Criterion 3 Criterion 4

Profiles Identification by Criteria - Thresholds of Indifference, Preference, Veto and Weights

The applied method defines two profiles (b1,2 ) for the set of four criteria. The decision to adopt only two profiles was suggested both by the low variability of the indicators used as criteria, and by the objective analytical and interpretive difficulty, related to the choice of a higher number of profiles. The choice of two profiles (h = 1, 2) automatically leads to the identification of three categories (we define Ch+1 the third and last category). The three categories represent the following conditions of aggregation: Category C1 : Measures with high risk Category C2 : Measures with intermediate risk Category C3 : Measures with low risk The profiles defined on the criteria gj (b1,2 ) have been associated with the thresholds of strict preference pj (b1,2 ), indifference qj (b1,2 ), veto vj (b1,2 ) and the weights wj . The conditions imposed on the system are as follows3 : gj (bh+1 ) ≥ gj (bh ) + pj (bh+1 ) + pj (bh ), ∀j ∈ F, ∀h ∈ B categories consistency pj (bh ) ≥ qj (bh ), ∀j ∈ F, ∀h ∈ B thresholds consistency qj (bh ) ≥ 0, pj (bh ) > 0, ∀j ∈ F, ∀h ∈ B constraints of non-negativity vj (bh ) = 0, ∀j ∈ F, ∀h ∈ B absence of veto thresholds wj = 1, ∀j ∈ F, weights invariance λ ∈ [0.5, 1] variation range of λ

From equation (1) to (3) it follows: gj (bh+1 ) > gj (bh ), Given the difficulty of establishing shared values of veto, a condition of absence of veto thresholds has been imposed4 . Similarly, a condition of absence of differentiation based on the weights was imposed, introducing a weight equal to one for all criteria5 . The two profiles gj (bh ), as well as the relative thresholds, have been defined taking into account: i) the results of the analysis on the statistical distributions of the variables used as criteria; ii) the considerations expressed by the intermediate evaluator and by the technical experts supporting the MA; iii) what emerged during the preparation of this work. In Table (2), the values assigned to the profiles and the thresholds according to the constraints listed above are reported. The value of the threshold λ was fixed in 0.8. The overall pattern of the two profiles is shown in graphical form (see Figure 3), with the caveat that the graphic representation of the criterion with decreasing preference has been made possible by changing sign to the values. The distribution of the values for each alternative (Measure) in the criteria, within the scheme of profiles and thresholds, allows us to assign the Measures to the categories, based on 3

The policy makers state all the conditions in the case of increasing preference. The case of decreasing preference, which is only found for the indicator ”number of changes of RM”, comes with the appropriate adjustments. 4 The hypothesis of absence of the veto thresholds, with wj = 0, ∀j ∈ F , determines that the index of discordance dj (a, bh ) can assume only the two extreme values: dj (a, bh ) = 1, or dj (a, bh ) = 0. 5 This hypothesis implies that the global concordance index is calculated as the simple average of the indices of partial correlation.

Table 2: Profiles, thresholds of indifference, preference, veto and weights per criterion MM Support Pay/Appr. Changes Profile 1 gJ (b1 ) qJ (b1 ) pJ (b1 ) vJ wJ gJ (b1 ) + pJ (b1 ) + pJ (b2 ) Profile 2 gJ (b2 ) qJ (b2 ) pJ (b2 ) vJ wJ Categories consistency Thresholds consistency Value of λ

the values of the indices of concordance (partial and total) and discordance, into two different scenarios: pessimistic (π) and optimistic (ω).

Figure 3: Graphical Representation of the Profiles and Intervals of Indifference and Preference

Once assigned the values of pj (bh ) and qj (bh ), the threshold λ affects the process of assigning the categories, given that σ(a, bh ) ≥ λ ⇒ aObh 6 . The three binary relations of preference ( or ≺), indifference (I) and non-comparability (R) are defined as follows: • aIbh ⇔ aObh and bh Oa • a  bh ⇔ aObh but not bh Oa • a ≺ bh ⇔ not aObh but bh Oa • aRbh ⇔ not aObh not bh Oa (not comparability) It should be noted that the less favourable outcome i.e the non-comparability in both profiles is present only in three out of 43 Measures. The non-comparability in both profiles (case of double R) is a condition that makes the assignment to the categories in the two proposed solutions (pessimistic and optimistic) unreliable. Overall, 21 category changes in the transition, from pessimistic scenario to the optimistic scenario, are identified, thus leaving unchanged the position of 22 Measures, slightly more than 50% of the program. As already noted above, the assignment to categories in both scenarios is strictly dependent on the value imposed to the threshold λ. For this reason, a sensitivity analysis was carried out aimed to evaluate the effect of a variation of λ on the distribution of the Measures between the classes. In particular, it was noted that - for values of 0.5 ≤ λ < 0.8 - only one Measure changes category and only in the case of pessimistic procedure (with transition from C3 to C2 ). For values of λ > 0.8, in a sufficiently small threshold around 0.8 (for example, by introducing an increase of 0.01, then with λ = 0.81), there is a shift of four Measures, three from C3 to C2 in pessimistic scenario and one from C2 to C3 in optimistic hypothesis. The observed movements are therefore quite limited and affect only marginally the results of the analysis. Moreover the group of Measures assigned to the category C1 (the worst category) remains constant. This category is representative of a high-risk conditions so it is also the most interesting for the choices of DMs.

Table (3) shows the distribution of the Measures in the three classes of risk associated with the two scenarios, ”pessimistic” and ”optimistic”. The analysis of the results shows a positive assessment of the program as a whole, the overall picture is obtained for the two scenarios in Table (3). In the pessimistic hypothesis, though the proportion of Measures in category C1 (high risk) is greater than 50%, some Measures with financial dimensions below average are also included, so that the percentage of the category, in terms of the total budget of the program, falls to 38%. On the contrary, the category C3 is composed by Measures with a financial value higher than the average (12% in terms of number of Measures, 24% in terms of appropriations). 6

In absence of the thresholds of veto, as in this case, the indicator of credibility has a direct relationship with the threshold of indifference and inverse with the threshold of preference.

Table 3: Distribution of Measures between classes of risk in the two procedures Number of Measures % on total (43) % of appropriations C1π High risk C2π Medium risk C3π Low risk C1ω High risk C2ω Medium risk C3ω Low risk

Pessimistic procedure π 23 53.49% 15 34.88% 5 11.63% Optimistic procedure ω 4 9.30% 27 62.79% 12 27.91%

In the optimistic hypothesis, Measures at high risk account for only 9% of the total and a similar percentage is found in the calculation of the appropriations (about 6%). The share of low-risk Measures grows, in comparison with the previous assumption, respectively to 28% and 38%. Five Measures, which fall in the category C3 in both solutions and represent almost a quarter (24%) of the total budget, can be judged as ”excellent”, while a high-risk condition is detectable only in three Measures because they are present in category C1 in both solutions. These three Measures represent only 6% of the total budget. In order to obtain a rough representation of the implicit weights associated to each criterion, a simple regression model was implemented. As stated above, the multicriteria model has been set up without any explicit determination of weights (i.e. wj = 1, ∀j ∈ F ) and this choice permits to estimate the ”natural” relevance (the implicit weight) that the criteria have in the explanation of the final classification within the set of alternatives. The first step in this analysis consisted in assigning a score to the three categories (C1 , C2 and C3 ) in the two alternative scenarios: pessimistic (π) and optimistic (ω). Considering that ∗gamma ∈ B being B = {1, 2}, where γ = 1 corresponds to the pessimistic scenario and γ = 2 to the optimistic one, then7 :

if σ(a, bh ) ≥ λ then C1γ = 3, 3 if σ(a, b1 ) ≥ λ and σ(a, b1 ) < λ and σ(b2 , a) < λ then C21 = 2, C22 = 3 if σ(a, b1 ) ≥ λ and σ(b2 , a) ≤ λ then C2γ = 2, 2 if σ(a, b1 ) < λ and σ(b1 , a) < λ and σ(b2 , a) ≤ λ then C21 = 1, C22 = 2 if σ(bh , a) ≥ λ then C3γ = 1, 1 if σ(a, b1 ) < λ and σ(bh , a) < λ non comparability

In the second step a new ordinal variable, named E, was defined, associating the score values of E with given combinations of Ch values, for each alternative (Measure) a, in pessimistic (π) and optimistic (ω) scenarios. New variable E ranges from a minimum of 2 to a maximum of 6, as the Table (4) below shows. 7

In the cases σ(ak , bh ) < λ and σ(bh , ak ) < λ a double R condition is verified.

Table 4: Values of E variable for each level of rank Ranking π ω E C1 Highest rank

A simple regression model (13) estimates the values of the implicit weights for each criterion gj , E = θ[gj (a)] then, assuming linear hypothesis, Ek = β0 +

With k ∈ A being A = {1, 2, ..., n}, where n = 40 (total number of alternatives, Measures, excluding three Measures falling in the hypothesis of non-comparability) and j ∈ F being F =< {1, 2, ..., m}, where m = 4 (number of no default criteria). Correlation matrix shows the presence of a possible collinearity in criterion MM (see Table (5)), so the results of the regression model, without criterion MM omitted, due to the collinearity, are presented in Table (6): Table 5: Correlation matrix between dependent variable E and four criteria E MM Support Pay./Appr. Changes E 1 MM 0.4579 Support 0.3769 Pay./Appr. 0.5772 Changes -0.5927

All the three criteria are significant in explaining the distribution of the Measures in the ranking expressed by the variable E. Implicit weights are relatively high for ”Number of officials assigned to support the MM” (criterion 2, with a coefficient of 0.368) and ”Numbers of changes in the position of MM” (criterion 4, with a coefficient of -0.283), rather low for the criterion ”Payments/Appropriations” (criterion 3, with a coefficient of 0.028). The organizational aspects inheriting the management of the Measures seem to be more effective in improving the quality level of the Measures.

Conclusion

The application of ELECTRE TRI method to the data of the Regional Operational Program has led to the assignment of the Measures into three categories: high Risk (C1 ), medium risk

Table 6: Estimation of criteria implicit weights. Dependent variable: ranking variable E Criteria Coefficients t values Support Pay/Appr. Changes Constant Number of obs R-squared Adj R-squared F (3, 36)

(C2 ), low risk (C3 ). The assignment in the three categories was done through two scenarios: pessimistic and optimistic. The program was therefore assessed through the distribution of the Measures in the categories for the two scenarios. This work had two objectives: i) on one hand, to ensure that the use of multi-criteria approach, in particular through the ELECTRE TRI method, could represent a possible methodological tool to evaluate complex programs, such as EU Structural Funds Program, in a systematic way; ii) on the other hand, to demonstrate that the application of the ELECTRE TRI method to a regional operational program, can get satisfactory results in terms of classification of the Measures and could improve the decision making process of stakeholders and policy makers. Regarding the first objective, it is possible to conclude that the ELECTRE TRI can be a useful tool to build up the programming choices but the evaluation targets must be clearly stated and a collection of specific information, integrated with the action of the evaluator and technical assistance must be developed. Other key aspects are: i) continuous contact with the responsible Managing Authority for the implementation of the program; ii) and a deep understanding of the needs of beneficiaries. Overall all the needed qualitative and quantitative information are actually available within the system of co-ordination, monitoring and evaluation of Community programs. This methodological approach does not propose any hierarchical ranking among Measures, but a structured classification of the same, thus allowing the DM to decide in more appropriate way, taking in consideration the political and territorial constraints that are often crucial during the implementation of a long-term and financially relevant program. In this study, no weights on criteria have been used, but no doubt, this could be an additional tool to further refine the results of processing: the choice of weights could largely simplify the introduction of the territorial and political constraints mentioned above. With regard to the second objective, the few observations on the analytical results proposed in the previous section show that the adoption of the method is an effective tool of interpretation and management. In a clear and concise way, the policy maker gets the information about how critical the Measures are so that he can adjust the classification just unveil which weights has been assigned to all criteria, on the basis of the policy directions that he considers most relevant. The

model can also be used to simulate the effects of policies aimed to improve the efficiency and effectiveness of the Program.

References Berk, R., and P. Rossi (1999): Thinking about Program Evaluation. SAGE Publications. Boccia, F., and et al (2003): I Mezzogiorni di Europa - Verso la riforma dei fondi strutturali. Il Mulino, Bologna. Brancati, R., and P. R. P. (2003): “Un approccio metodologico per la valutazione intermedia: il caso del Programma Operativo della Campania 2000-2006,” cienze Regionali, 3, 29–56. Chen, T. H. (1990): Theory-driven Evaluation. SAGE Publications, Newbury Park. (2004): “The Roots and Growth of Theory-Driven Evaluation: An Integrated Perspective for Assessing Viability, Effectuality, and Transferability,” in Evaluation roots, ed. by M. C. Alkin. SAGE Publications, London. (2005): Practical Program Evaluation. SAGE Publications, London. Cronbach, L. J. (1982): Designing evaluations of educational and social programs. JosseyBass, San Francisco. Dias, L., and J. Clı́maco (2000): “ELECTRE TRI for Groups with Imprecise Information on Parameter Values,” Group Decision and Negotiation, 9(5), 355–377. European Economic Community (1987): European Single Act. J L 169 of 29.6.1987, Luxembourg. European Union (1992): Treaty on European Union (Maastricht Treaty). OJ C 191 of 29.7.1992: Office for Official Publications of the European Communities, Luxembourg. Figueira, J., and B. Roy (2002): “Determining the weights of criteria in the ELECTRE type methods with a revised Simos’ procedure,” European Journal of Operational Research, 139(2), 317–326. Liesbet, H., and G. Marks (2001): Multi-Level Governance and European Integration. Rowman and Littlefield, Buolder. Lourenço, R. P., and J. P. Costa (2004): “Using {ELECTRE} {TRI} outranking method to sort {MOMILP} nondominated solutions,” European Journal of Operational Research, 153(2), 271 – 289. Marks, G., L. Hooghe, and K. Blank (1996): “European Integration from the 1980s: State-Centric v. Multi-level Governance*,” JCMS: Journal of Common Market Studies, 34(3), 341–378.

Mayntz, R. (1999): “La teoria della governance: sfide e prospettive,” Rivista italiana di scienza politica, 1(3), 3–21. Ministero del Tesoro (1999): Programma di sviluppo del Mezzogiorno. Donzelli, Roma. (2000): del Bilancio e della programmazione Economica. La nuova programmazione e il Mezzogiorno. Donzelli, Roma. Monti, L. (1996): I fondi strutturali per la coesione europea. SEAM, Roma. Mousseau, V., and R. Slowinski (1998): “Inferring an ELECTRE TRI Model from Assignment Examples,” Journal of Global Optimization, 12(2), 157–174. Owen, J. (2006): Program evaluation - Forms and approaches. SAGE Publications, London. Rogers, M., and M. Bruen (1998): “Choosing realistic values of indifference, preference and veto thresholds for use with environmental criteria within {ELECTRE},” European Journal of Operational Research, 107(3), 542 – 551. Rossi, P. H., E. H. Freeman, and M. W. Lipsey (1999): Evaluation - a systematic approach. SAGE Publications, London. Roy, B., and D. Bouyssou (1993): Aide multicritere a la decision: Methodes et cas. Economica, Paris. Roy, B., and V. Mousseau (1996): “A Theoretical Framework for Analysing the Notion of Relative Importance of Criteria,” Journal of Multi-Criteria Decision Analysis, 5(2), 145–159. Scharpf, F. W. (1994): “Community and Autonomy Multilevel Policy-Making in the European Union,” Discussion Paper 1, European University Institute (EUI), Robert Schuman Centre of Advanced Studies (RSCAS). The European Commission (2004): A new partnership for cohesion convergence competitiveness cooperation - Third report on economic and social cohesion. The European Commission, Luxembourg. Unione Europea (2000): Quadro Comunitario di Sostegno per le Regioni Italiane. Decisione C(2000) 2050 del 01.08.2000, GUCE L186/4 del 17-7-2002, Bruxelles. Weiss, C. H. (1972): Evaluation. Prentice Hall, 1972, New Jersey.

Share and Cite