|Main authors:||Janja Rudolf, Špela Železnikar, Matjaž Glavan, Andrej Udovč, Sindre Langaas, Marina Pintar|
|Source document:||»Rudolf, J. et al. (2021) Actor's feedback on practices for improvement of water quality in FAIRWAY case studies and interim project results. FAIRWAY Project Deliverable 7.2R 74 pp|
|1. Questionnaire and survey methodology|
|2. Responses to the questions|
|3. Conclusions on FAIRWAY interim results|
Responses from questionnaires completed by participants at the Joint FAIRWAY/WATERPROTECT Policy conference held in Brussels on 7th of December 2018 (53 attendants and 12 fully completed paper questionnaires) were supplemented with the use of CAWI method (Computer-Assisted Web Interviewing), which is a data gathering via the internet. A link to the web questionnaire was sent to all stakeholders that have accepted to be on the FAIRWAY mailing list for receiving information or invitations to an event. This list has also been used for the invitation for the Join Policy conference. The link to the web questionnaire of 306 mail addresses was sent on 27th of February 2019 with eight days to finish the web questionnaire. In this period, no other type of emails or other ways were used to increase the number of received questionnaires. After the due date, we received 23 questionnaires, of which 11 were fully finished (48% completion rate), but we were able to use 17 of 23. For the full analysis, we took answers of both the paper and web questionnaires (total of 29 fully finished questionnaires) and analysed them together.
Completion rates lower than 60 % in web questionnaires, should be examined for possible major errors in the survey design or logic (Liu and Wronski, 2018). The duration of the questionnaire was approx. 10 minutes. This is a questionnaire with long duration (10 – 15 minutes) according to Trouteaud (2004), and the completion rate of the questionnaire with this duration is statistically lower than in questionnaires with a short duration (3 – 5 minutes). This questionnaire is considered as more complex, as it contains open-ended questions and multiple choice questions with many words. The inclusion of difficult questions reduces a survey’s participation rate and increases the chances of respondents engaging in undesirable survey practices, such as item nonresponse or the use of heuristics like straight-lining (Liu and Wronski, 2018). The survey literature has shown that some survey formats are inherently more challenging to respond to than others. For example, a lengthy question imposes both comprehension and mapping difficulties (Holbrook, Cho, & Johnson, 2006). Also, open-ended questions are typically associated with higher dropout rates because they are more burdensome and require higher cognitive efforts than closed-ended questions (Manfreda & Vehovar, 2002). In a mail survey experiment, Dillman, Sinclair, and Clark (1993) found that the completion rate decreased when a difficult question was asked.
The primary goal of the survey was to gain suggestions for direction and improvement of FAIRWAY and obtain a measure of the quality of interim findings of the project. This means that the number of responses is meaningful, even with lower completion rates (Archer, 2008). The web questionnaire was designed in the free licensed Slovene web program 1KA (www.1ka.si), which is specialised in constructing web surveys of this kind.
The questionnaire combined open-ended questions and multiple-choice questions with predefined answers offering respondents the possibility to choose and/or rank among several options or the possibility to grade on a “very low” to “very high” scale. The questionnaire had four blocks. The blocks were divided according to the themes of the work packages.
In the questionnaire held in Brussels, we designed some explanatory questions, asking respondents to explain their opinion on matters concerning the interim findings of the work package on Science and Policy Support. Analysis of answers showed that they often repeated, a decision was made that for the web questionnaire the explanatory questions will be upgraded into multiple choice questions were the section Other (for other opinions 27 different than proposed) was permitted.
- In Block 1 we used questions with a Likert scale for the interim findings of the work packages on »Monitoring and Indicators, »Farming Practices, »Decision Support Tools and »Policy and Governance. Questions in the survey and web survey were based according to the Likert scale from 1 to 7, where 1 meant not useful, and 7 meant very useful to the respondent. Respondents were asked to put their choice according to the Likert scale proposed in the instructions. In the analysis of the results, we presented an average of Likert scale for each interim finding for each work package. Standard error and coefficient of variation (CV) were calculated.
- Blocks 2 and 3 contained questions relating to the work package on »Science & Policy Support. We wanted to know if the respondents agree with a solution (which was explained through the sketch in survey and e-survey) that the lead partners of the work package proposed as an answer to the interim findings. Only close-ended questions were proposed. In addition, the respondents were asked if they would choose other solutions, and if so, which solution.
- Block 4 asked which ways of communicating and disseminating the results are of most use to the respondents.
The full questionnaire is given in the Annex of this report:
In total, 306 experts, actors, policy-makers, farmers, non-governmental organisations, small and medium-sized enterprises and big companies from European Union countries were selected to conduct the Survey. They received an invitation to attend the Joint Policy Conference (JPC) held in Brussels and 53 of them respond and attend. At the conference, the paper questionnaires were distributed among all attendants. After the conference, the decision was made by WP 7 lead partner (UNI LJ) to send a link to an upgraded web questionnaire to all stakeholders that first received an invitation mail to join the JPC.
The stakeholders were selected by their field of expertise in water policy regulations/protection and the Pesticides and Nitrate Directives of EU or field of involvement in protection/pollution of EU water resources or integrated life within water protection areas.
The majority of respondents were from research institution (27 %), followed by the industry sector (20 %), regional institution (13 %), EU commission (13 %), national institutions (10 %), NGO (10 %) and 7 % respondents from SMEs. From the industry sector, the respondents defined their enterprises as fertiliser company, water supply company and pesticides industry. None respondent came from the stakeholder group – farmer (Figure 15).
How useful are the interim findings of WPs 3-6 to the respondents?
In this block, the interim findings of four of the work packages were presented in sentences. The respondents were asked to state the usefulness of these findings with the help of a Likert scale from 1 to 7, where one meant not useful, and seven meant very useful. The answers were collected from survey and e-survey and combined, and the arithmetic mean was calculated. Standard error and CV was calculated for each of the interim findings of each of the work package. The interim findings are presented below, and the results are presented in the paragraph.
WP3 Monitoring & indicators (WP3)
- Interim finding WP3_a: The most critical pressure indicators for the quality of drinking water on farms depend on the type of catchment.
- Interim finding WP3_b: Some link between pressure indicators and states indicators can statistically be performed.
Farming Practices (WP4)
- Interim finding WP4_a: There are many possible measures to decrease the pesticides pollution of drinking water supplies. Most effective measures are (i) spray drift reduction through technical modifications of the spraying technique, (ii) pesticides input reduction through integrated pest management measures, (iii) no spraying zones and vegetated buffer zones, and (iv) erosion reduction measures. Tillage measures appear to have little effect.
- Interim finding WP4_b: There are many possible measures to decrease the nitrate pollution of drinking water supplies. Most effective measures are (i) nitrogen input control, (ii) adjustment of crop type and/or crop rotation, (iii) growth of cover crops, (iv) minimum tillage and surface mulching, and (v) nitrification inhibitors. Fertiliser type appears to have little influence, while the effectiveness of buffer strips greatly depends on soil and hydrological conditions.
- Interim finding WP4_c: The estimated costs greatly vary between measures and also between countries. Some measures are cost-effective. Accurate cost information is scarce.
Decision Support Tools (WP5)
- Interim finding WP5_a: All participating countries have their own decision support tools (DSTs) developed to support water quality/agri/environment policy makers operating at a regional or national level, and those intended to support sustainable nutrient management at the farm level.
- Interim finding WP5_b: Only a few of the evaluated DSTs, evaluated at FairWay project, are primarily aimed at improving water quality. Instead, they are a farm (nutrient/pesticide) management tools based on the assumption that the efficient use of nitrogen and pesticides indirectly improves water quality. Only a few DSTs consider the impact of mitigation methods on water quality.
- Interim finding WP5_c: Decision support tools are not easily transferred from one country to another because they all operate within the context of the more comprehensive advisory frameworks in place in 30 their respective countries, in addition to issues around language and requirements for country/specific data, calibration, etc.
Policy & Governance (WP6)
- Interim finding WP6_a: The regulatory structures in all countries are very comprehensive and fragmented, to the extent that stakeholders are not able to fully understand them.
- Interim finding WP6_b: The governance structures between countries have considerable differences. Partly explanation lays in historical, cultural and political differences between countries.
- Interim finding WP6_c: Between countries, it is a high degree of divergence in scales of governance.
Results show that most findings were considered as useful to strongly useful (Figure 16). However, neutral reaction to interim findings is also quite strong. The most useful interim findings are WP5_b and WP6_a, where only 3 and four respondents (out of 25) decided that these findings are neutral or not useful to them.
The majority of answers has an average of Likert scale (Figure 17) between 70 and 74 %. The interim findings of the FAIRWAY project’s WP 3 to 6 are considered as at least slightly useful to the majority of the respondents. However, the average is higher when the findings are more precise, not so general, and therefore ready for further consideration.
How useful are the interim findings of WP 7 (Science & Policy Support) to the respondents?
In »Barriers and issues in providing integrated scientific support for EU policy we discussed with EU members barriers and issues concerning low interaction of project dissemination between researchers and policymakers in the EU.
- Interim finding WP7_a: Results show that EU research project dissemination is not followed through to the European Commission.
We wanted to know the opinion of respondents, the reason why this occurs, and their suggestion for improvement. In the questionnaire on paper, the questions were open-ended and explanatory, that means we gave respondents the freedom to express their opinion, without any forewarning possible answers. During the analysis of the answers, it was shown that answers were repeated and can be put together into common points. Therefore, we upgraded the explanatory questions into multiple choice questions for web questionnaire and added the section Other, for letting express their opinion if it could not be put into any of suggested common points. However, the analysis of the web questionnaire results showed that all the answers could be still put together within the common points. The results are shown below with the paragraph.
The most frequently cited answer was "Complex governance system where key measures are easily lost" (cited ten times), followed by "Often to academic terminology" (cited eight times) and "Not well communicated" and "Not sufficiently bottom-up approach" (both cited seven times). These answers were the crucial issues that respondents recognise as problematic for inefficient project dissemination followed through to EU (Figure 18).
Almost all of the answers offered were equally selected among respondents, which suggests that the respondents recognised solving these issues in multiple ways and on multiple scales. The cited answers range from 5 to 8, showing that solutions cited eight and seven times, could be preferred and solutions cited 6 and five times, could be supplementary for improvement of project dissemination efficiency (Figure 19).
- Interim finding WP7_b The need is to have key and important final project results shorter and in a language understandable to policymakers.
Respondents were asked whether they did not agree (1) or very much agreed (7) with this statement. It can be seen from Figure 20, that the structure of responses is undoubtedly pointed to the right, where most of the respondents chose that they agree or strongly agree with the statement.
The average of Likert scale was very high 6.2, which meant that most of the respondents strongly agree with this interim finding. This is also our most important conclusion and it points out the absolute need to have key and important final project results shorter and in a language understandable to policymakers.
- Interim finding WP7_c Some research projects focus on findings and fulfilling the Grant Agreement obligations, disregard whether the topic is on the political agenda.
To make the research projects more connected to the political agenda, the European Commission could establish Task forces with the aim of designing project clusters. This proposal is for a unique type of long-term relationship/communication flows in issues concerning quality of drinking water that is presented in the scheme below and was presented in questionnaires to all respondents (Figure 21).
We asked respondents if they think this could be a good suggestion for solving the “gap” between science and policy. Only close-ended question with answering yes or no was possible. The analyse show that 86 % of respondents find this solution as good. Next, we also asked which solution would be better for solving the “gap” between science and policy, and here multiple choice questions were proposed, also with the section Other, but none has used it. The results are shown in the paragraph below.
The solutions: "Through various events" and "Open communication flow between DG AGRI and DG ENVI" were most cited (8 times) and are thus the preferred solutions. The other two (cited 6 and 5 times), can be considered as supplementary solutions (Figure 22).
Which ways of communicating and disseminating the results are of most use to the respondents?
The respondents agree that it is the best way to receive the interim findings of the project via conference/workshops or executive summaries of deliverables. The second best way is via short media news like YouTube channel, short policy briefs and subscription to the Newsletter. Field visits were an additional suggestion from one respondent answering the web questionnaire. This suggestion was not presented to all respondents to choose, and therefore, it cannot be concluded as the most unattractive way for respondents to receive interim results of the project, as suggested in Figure 23.
The best ways of communication and dissemination of the final results of the project are executive summaries of deliverables, followed by conference/workshops, articles in scientific journals and YouTube videos (Figure 24). These findings can serve for further development of the FAIRWAY deliverables in order to help distribute the findings of the project to the respondents within the most effective communication channel. For field visits, the same comments hold as for Figure 23.
The while communication channels are important, so is communication style and it should be discussed further. We know that there are some useful communication styles that work better between science and policy. Safford and Brown (2019) discussed this issue in their article called Communicating science to policymakers: six strategies for success. We feel it is in place that we shortly introduce their work:
6 Strategies for success (Safford and Brown, 2019):
- Know who you want to reach. If you are not sure who you need to reach, ask around!
- Have clear and actionable recommendations. Your suggestions should be feasible. Every government body is constrained by its mission and budget. Do your best to propose actions that fall within your target agency’s authority.
- Repackage your work. The peer-reviewed article is the currency of the scientific realm, but it is not going to get you far in policy. A new audience demands a new format — one that is accessible and understandable.
- Write well. Conversations and presentations are great ways to introduce a topic, but policymakers will want a written product to react to or to share with colleagues. Organization, brevity and clarity are more important than wit or style when it comes to policy writing. State your key points first, then provide more explanation. Make sure there is a clear one-sentence takeaway in the very first paragraph.
- Pick your moment. Strategically selecting when to engage increases the chance that your idea will fall on receptive ears. Electoral and legislative calendars can help you to choose a good time. Meetings with elected officials tend to be much more effective towards the beginning of a term (when policy priorities are being set) than towards the end.
- Sustain and amplify your engagement. Building support takes time and ongoing effort. Collaborating with people and institutions who have an agenda like yours is a great way to strengthen your collective case.
Some of the greatest weaknesses in communication style between science and policy in water policy issues that were observed among most respondents were (Fig. 18):
- complex governance system where key measures are easily lost,
- often terminology is too academic,
- not well communicated and
- not sufficiently bottom-up approach.
These answers were the crucial issues that respondents recognise as problematic for inefficient project dissemination followed through to EU and should be concerned further. If we look the 6 strategies for success and the results coming from the survey we can see that we have most problems in using strategies 1, 2, 3 and 4 (Safford and Brown, 2019). Table 2 presents weaknesses and strategies to solve this weaknesses by Safford and Brown (2019).
Table 2: Recognized weaknesses in communication style between science and policy and strategies to solve these weaknesses by Safford and Brown (2019)
|Weakness||Strategy by Safford and Brown, 2019|
|Complex governance system where key measures are easily lost.||Know who you want to reach. If you are not sure who you need to reach, ask around!|
|Often terminology is too academic.||Repackage your work. The peer-reviewed article is the currency of the scientific realm, but it is not going to get you far in policy. A new audience demands a new format — one that is accessible and understandable.|
|Not sufficiently bottom-up approach.||Have clear and actionable recommendations. Your suggestions should be feasible. Every government body is constrained by its mission and budget. Do your best to propose actions that fall within your target agency’s authority.|
|Not well communicated.||Write well. Conversations and presentations are great ways to introduce a topic, but policymakers will want a written product to react to or to share with colleagues.|
To ensure that the findings of the project will follow through to the policy makers scientists need to combine the knowledge of which communication channels are more appropriate for stakeholders and to properly use the right communication style to improve on informing politicians and other policymakers on how to make decisions.
Our primary goal of the survey was to gain suggestions for direction and improvement and obtain a measure of the quality of interim findings of the FAIRWAY project. The stakeholders were selected by their field of expertise in water policy regulations/protection and the Pesticides and Nitrate Directives of EU or field of involvement in protection/pollution of EU water resources or integrated life within water protection areas. Results show that the most useful interim findings of WP 3 to WP 6 were:
- Only a few of the DSTs evaluated in the FAIRWAY project are primarily aimed at improving water quality. Rather they are a farm (nutrient/pesticide) management tools based on the assumption that the efficient use of nitrogen and pesticides indirectly improves water quality. Only a few DSTs consider the impact of mitigation methods on water quality.
- The regulatory structures in all countries are very comprehensive and fragmented, to the extent that stakeholders are not able to fully understand them.
The majority of responses on the Likert scale ranged between slightly useful to useful. This shows that the interim findings of the FARIWAY project’s WP 3 to 6 are useful to the respondents. However, the average of the Likert scale is higher when the findings are more precise, not so general, and therefore ready for further consideration.
The respondents recognised the following issues as most problematic for inefficient projects dissemination followed through to EU:
- Complex governance system where key measures are easily lost,
- Often too academic terminology,
- Not well communicated and
- Not sufficiently bottom-up approach.
Almost all of the answers offered for solving the problem of inefficient projects dissemination are equally selected among respondents, which suggests that the respondents recognise solving these issues in multiple ways and on multiple scales.
One of our most essential conclusions is that there is an absolute need to have a short key summary and important final project results and in a language understandable to policymakers. Most respondents decided that they strongly agree with this statement. The average of Likert scale was very high, 6.16 out of 7. The number of responses for this statement was 25.
We presented a scheme of possible long-term relationship/communication flows between research projects and political agenda to respondents. The analysis showed that 86 % of respondents agreed with this solution.
Finally, the respondents were also asked how they like to receive interim and final project’s findings. The respondents agreed that the most effective way to receive the interim findings of the project is presentations at a conference/workshops or via executive summaries of deliverables. The final results of the project can be best communicated via executive summaries of deliverables, and secondly conference/workshops, articles in scientific journals and YouTube videos.
In the end, it should be explained that this gathered data is highly appreciated for the project findings and will help in many ways with further research. The studied samples in both questionnaires were small (30 and 29 respondents). Therefore, larger samples of respondents at EU level or inclusion of local level stakeholder groups not included in the project (MAPs) could impact on the result.
Note: For full references to papers quoted in this article see