The implementation of a public policy (see intra) refers to the performance of activities which, by using a certain amount of resources, aim to achieve certain results. Policy implementation analyses (or process evaluation) is intended to reconstruct "what happened" in the implementation process and to check the presence of any implementation deficits that might undermine the policy results. Unlike the effects evaluation, which aims to estimate whether the policy was able to transform reality in the desired way, implementation analysis focuses on how the policy functions. It checks, for example, whether the administrative procedures are appropriate, practitioners' ability to make themselves understood by beneficiaries, communication quality, the internal dynamics of the administrations involved, funding delivery mechanisms, etc.

Two fundamental approaches to the implementation process feature widely in the literature. The first (top-down) observes the process from the decision-makers' perspective. It takes as its benchmark the policy goals established when the policy itself was approved and interprets as an implementation deficit any and all differences between the initial goals and the results actually achieved. The second approach (bottom-up) observes the implementation process "from the ground up". It takes into consideration the attitudes, motivations and reactions of the intended beneficiaries of the policy actions and of their immediate contact persons, i.e. the practitioners directly involved in the policies (for example teachers, public-facing counter staff, police officers, etc.).

Sources:

- Martini A. and Sisti M., Valutare il successo delle politiche pubbliche, Bologna 2009, p. 30 et seq.;

- World Bank, Impact Evaluation in Practice, Washington 2011, p. 233;

- La Spina A. and Espa E., Analisi e valutazione delle politiche pubbliche, Bologna 2011, p. 64 et seq.;

- Associazione per lo Sviluppo della Valutazione e l'Analisi delle Politiche Pubbliche (ASVAPP) slides on Implementation analysis and logic model (Valentina Battiloro) as part of her Master's.

Internal validity refers to the validity of the sample of individuals used as the control group to estimate the counterfactual.

Internal validity is a necessary - but not in itself sufficient - condition for a study to be useful.

External validity refers to the possibility of generalising the conclusions drawn from an evaluation to the entire population of individuals who are eligible for a policy. For sample to have external validity, it must be representative of the population (which in general is not known).

Interrupted time series analysis is a method of evaluating non-experimental effects which exploits the temporal discontinuity presented by the "treatment", i.e. by the policy being evaluated.

It is mainly applied to universal policies, i.e. those involving the entire reference population, which enter into force from a given date onwards (for example, the introduction of a new obligation or of a more severe penalty). In these situations, there are no individuals not "treated" by the policy so a contrafactual scenario cannot be constructed.

The method consists of predicting the performance which the result variable would have had in the period following the introduction of the policy if it had continued in its previous behaviour, thus simulating a counterfactual scenario.

The effect of the policy is estimated as the difference between the value observed (factual) and the value predicted (counterfactual) for the result variable.

For the estimates to be reliable, a time series of data for the result variable that embraces a sufficiently long timespan prior to the intervention must be available.