A variety of possible methodologies were explored to assess the level of vertical and horizontal coherence, including the use of external expertise. The task, however, requires a high level of understanding of EU legal directives and policies, a breadth of knowledge and perspectives, and a variety of views from respondents in different roles and positions. The FAIRWAY partners judged the method described below to be the most appropriate for the task as it ensures the representation of a wide number of perspectives across sectors and scales, from different geographical areas in Europe, and wide variety in roles and positions. The FAIRWAY partners represent 13 different European countries. Using in-house expertise, rather than involving external consultancy expertise, also entailed an advantage in terms of understanding the purpose of the research and task and the existence of prior knowledge with regard to the various directives. This enabled a cost-effective and sound methodology to complete the task.

Four step coherence assessment methodology

The assessment of vertical coherence, i.e. the contribution of the legal framework towards the overarching aim of protecting drinking water resources, and horizontal coherence, i.e. the cohesion amongst the various requirements and directives, consists of several steps.

Table 1.3 Steps towards a coherence assessment of EU legal requirements

Step 1 – Inventory of all requirements and objectives

In a first step, we identified the key requirements and objectives of the various directives and policies. The purpose of the inventory step was to get a comprehensive overview of the requirements and objectives of all instruments. This is a descriptive analytical task.

We designed a template for the review of the relevant instruments. The template distinguished between different categories of requirements (ecological requirements, and requirements related to reporting and monitoring, public participations, and coordination). The template was discussed and tested out in September 2017 for the Water Framework Directive. The template was slightly adjusted and then approved by the group of actors, consisting of ten of the partners to the FAIRWAY project.

After this, the instruments were divided among the partners. Sub-groups were established consisting of two participants from different partners. Together these participants reviewed the instrument designated to them. This review took place in the period between October-November 2017.

At the annual meeting for the FAIRWAY project in Naples (23-24 November 2017), a special session was devoted to this task. In this session, the group of participants carried out an additional check of the completed reviews to assure its correctness, completeness and quality. Each review was re-assessed by a new subgroup of two participants.

The final reviews are compiled in and presented in Appendix I of the full report:

Step 2 – Screening matrix

In a second step, we created a screening matrix that displayed all the different ecological requirements and objectives in Excel spreadsheets. We created different matrices: the first matrix displayed all ecological (including drinking water quality) requirements and objectives from the ten directives in relation to the overarching FAIRWAY objective to enable a vertical coherence assessment. A further five matrices were developed to display the requirements of five individual directives on the vertical axis against the requirements and objectives of other directives on the horizontal axis for the purpose of a horizontal coherence assessment. These five matrices focused on the most central directives; the Water Framework Directive, the Groundwater Directive, the Drinking Water Directive, the Pesticides Directive and the Nitrates Directive. For the purpose of screening and scoring, we focused primarily on the ecological and environmental requirements and objectives of the directives.

Based on the results from the vertical coherence assessment, five directives were identified as highly relevant for the attainment of the FAIRWAY objective. For that reason, the horizontal coherence assessment analyses these five directives thoroughly. The delimitation to these five directives, enabled a more thorough and in-depth horizontal coherence assessment than what would be possible if all directives had been included in this final analysis.

Step 3 – Scoring

In the third step, we evaluated and scored the contribution of the various legal requirements towards the overarching FAIRWAY objective using six online surveys. Survey One investigated participant opinions about the contribution of directives to the FAIRWAY objective. This survey was based on the first screening matrix developed in Step 2 (above). We distributed Survey One to ten participants during the period of March-April 2019. Survey’s Two-Six were based on the five specific matrices developed in Step 2, each addressing horizontal coherence amongst the legal requirements of the most central directives. These surveys were completed by five partners over the same time period. The surveys were distributed, mostly, in accordance with the partners’ involvement in the review process in 2017. For example, the partners who contributed the most to the review process for the WFD in 2017 were given the survey focusing on the coherence between the WFD and other directives. This ensured that the partners, as far as possible, assessed and scored the legal directive within their main field of expertise. Some partners have called on additional expertise of those working in the industry.

The partners have completed one survey each. The scores were generated based on an internal elicitation within the partner institutions. In most cases at least two individuals discussed a given interaction and provided their assessment of what the score ‘should be’.

All surveys included two types of items; quantitative Likert-scale items and qualitative open-ended items. The quantitative items asked participants to give a numeric score representing their perception about the interaction of a directive with either the overarching aim of FAIRWAY (Survey One), or with other directives (Survey Two-Six). Survey One contained 53 of these quantitative items. Each item addressed a different specific requirement of the target directives. For example, 9 items were included on Survey One to measure partner opinions about interactions between 9 requirements of the WFD and the overarching FAIRWAY objectives. These quantitative items were scored on a seven-point Likert-Scale from -3 to +3. The scale was based on the typology and seven-point scale presented by Nilsson et al (2016) to assess the degree of coherence.13 Pursuant to the seven-point scale, interactions may be scored as either positive (indivisible’ (+3), ‘reinforcing’ (+2) or ‘enabling’ (+1)) or negative (‘cancelling’ (-3)’, ‘counteracting’ (-2) or ‘Constraining’ (-1)); or the respective legal requirements may be entirely ‘neutral’ (0) with each other, incurring no significant positive or negative interactions whatsoever, perhaps no interaction at all.

Table 1. 4 Seven-point scale scoring based on Nilsson et al (2016)

Each survey also contained open-ended survey items to help interpret the quantitative data. These items asked respondents to give their opinion about the scorings. For example, open-ended items in Survey Two-Six asked respondents to explain their scorings and give examples.

Step 4 – Data analysis

In Step 4 we analysed the data, including quantitative and qualitative analysis. The scores from Survey One assessing vertical coherence between the directives and the overarching aim to protect drinking water resources against agricultural pollution were analysed by computing averages. Each survey item was given a score on the 7-point Likert-scale by ten separate respondents. For each item we computed the average value.

In contrast, only one partner completed each of the five surveys about the interaction between individual directives. Each survey compared the requirements of one key directive to multiple requirements of each other key directive. For example, in Survey Two, a partner gave scores about the interaction between 4 requirements of the WFD and 4 requirements of the GWD. To analyse this data, we considered each individual interaction, as well as computing the average score of interactions between all GWD requirements and each single WFD requirement. This approach enabled us to identify on average which directives had the most positive and most negative interactions with each other.

We also conducted a qualitative analysis of respondent answers to open-ended items in each survey. To do this we evaluated the key themes in each response and considered the frequency of occurrence of each theme. This approach allowed us to highlight critical areas where better understanding is needed. Thus, we produced a summary of potential challenging interactions that appear somehow uncertain or are subject to diverging views, and therefore are worthy of further investigation.

Limitations of the methodology

The assessments of the degree of coherence between the directives and FAIRWAY objective (Survey One), and between individual directives (Survey Two-Six) outlined in this report are based upon respondents’ perceptions and opinions. As such, some bias in the scorings and explanations is unavoidable. The horizontal coherence assessment (Survey One) was carried out by ten FAIRWAY partners. The five vertical coherence assessments (Survey Two-Six) have been divided among the partners specifically, for budgetary reasons. Given that each survey (for the WFD, GWD, DWD, ND and PD) has been carried out by one partner, this might affect the scoring rates. To increase accuracy of scoring rates, the surveys have been distributed in accordance with the partners’ main fields of expertise. Despite the delimitations, the methodology applied in the research offers great value; it contains an analysis of perceptions at a more privileged level of knowledge than usually available through literature review only.

Note: For full references to papers quoted in this article see

» References