Strasbourg, 23 October 2023

                                                                                                            CEPEJ-GT-EVAL(2023)11

EUROPEAN COMMISSION FOR THE EFFICIENCY OF JUSTICE

(CEPEJ)

WORKING GROUP ON THE EVALUATION OF JUDICIAL SYSTEMS

(CEPEJ-GT-EVAL)

45^th meeting

17-19 October 2023

(Annex IV to this document contains a summary of the discussions held at the 17^th meeting of CEPEJ national correspondents - Strasbourg, 18 October 2023).

MEETING REPORT

Document prepared by the Secretariat

Directorate General I - Human Rights and Rule of Law

I.                 ADOPTION OF THE AGENDA

1.             The Working Group on the Evaluation of Judicial Systems (CEPEJ-GT-EVAL) of the European Commission for the Efficiency of Justice (CEPEJ) held its 45^th meeting on 17 and 19 October 2023, in Strasbourg, under the chairmanship of Jaša Vrabec (Slovenia). 

2.             The agenda and list of participants at the meeting are given in Annexes I and II respectively to this report.

II.               2024 EVALUATION CYCLE (2022 DATA)

a.                Status and questions from the data collection and quality control

3.            The CEPEJ Secretariat informs the Working Group of the state of progress of the data collection process and the quality control of the data already provided by the national correspondents. The questionnaire was opened last March, and data collection was due to end on 1 October. To date, 44 Member States (with the exception of Liechtenstein and San Marino) and two observer states (Israel and Morocco) are taking part in this evaluation cycle.

4.            The Group discussed ways of improving data collection timeframes and the involvement of national correspondents. These included the importance of promoting an "institutional memory" in each country, given the regular changes in national correspondents and, more generally, of the internal organisation of national institutions, and the need for each country to have its own internal data control procedure.

b.                Data collection support: FAQs for national correspondents, specific support and peer reviews

5.             The new FAQ (frequently asked questions) section for national correspondents available on the CEPEJ website, which enables new interpretations to be given to questions or technical advice and instructions to be given to national correspondents during the evaluation cycle, was presented by the Secretariat. These questions/answers will also form a basis for future discussions on the revision of the questionnaire and the explanatory note.

6.             The Working Group was also informed of the progress of the data collection support missions organised in Greece and the Republic of Moldova, in January and September 2023 respectively.

7.             It was also agreed that the Secretariat would prepare a "Welcome package" containing all relevant information for national correspondents, in particular newly appointed correspondents, to enable them to better understand the procedure, their role and their tasks.

8.             More generally, the Group stresses the importance of training national correspondents and ways of strengthening their positions at national level in order to facilitate data collection.

c.                Structure of the report

9.             The Secretariat recalls that a questionnaire for the evaluation of the report was sent to CEPEJ members and national correspondents in June 2023. The Secretariat's analysis of the replies was used to prepare a draft structure for the next report. It was also presented to the Working Group and will be published shortly on the CEPEJ website.

10.          Concerning the next CEPEJ evaluation report, the Group decided to keep the three existing parts of the report: analysis of European trends, country profiles and CEPEJ-STAT database. Each of these parts will be updated in the light of the revised questionnaire, with some questions deleted and others added.

11.          With regard to the first part of the report, the Group agrees to :

-       devote Chapter 2 solely to budgets (courts, public prosecution services, legal aid, the judicial system as a whole) and move the analyses concerning other aspects of legal aid in particular to Chapter 4 ;

-       devote Chapter 3 to professionals; this chapter will have to include new developments with regard to the new issues of this cycle concerning, in particular, part-time work or the recruitment and promotion of judges and prosecutors, training and ethics;

-       amend Chapter 4, which is made up of sub-chapters on courts/users/new technologies (ICT), to incorporate developments relating to courts/users and legal aid, and rename it "Access to justice for users";

-       reorient the developments of Chapter 5 devoted to the efficiency of the courts on the CEPEJ performance indicators (clearance rate and disposition time) and develop, if possible, on the basis of the data collected, the analyses of the data relating to the prosecution services;

-       develop a new chapter on information and communication technologies (ICT) based on the new questions and presenting, if possible, not only the rate of deployment but also the rate of use of these technologies.

12.          More generally, the experts would like the report to contain data for a 10-year period (data from 2012 to 2022) and for the Crystal Scales of Justice projects containing national justice initiatives to be given greater prominence. Some chapters could also contain analyses and graphical representations of groups of countries showing increases/decreases, for example in terms of budget.

13.          As regards the country profiles, the Group agreed to keep the same content (analyses and indicators) as in the previous cycle, but to present them on four pages instead of three to make them easier to read.

14.          With regard to CEPEJ-STAT, it was agreed to improve navigation for users. A discussion on a possible new dashboard could take place at the next CEPEJ-GT-EVAL meeting once most of the data has been collected and stabilised.

d.                Timetable of the report

15.          The data collected and checked by the Secretariat ("quality check") will be stabilised at the end of February and submitted to the experts responsible for drafting the report by 1 March 2024 at the latest. The data will be accompanied by comments from the national correspondents and statistical material prepared by the Secretariat. The final versions of the draft chapters should be sent to the Secretariat at the beginning of May. The Secretariat will revise these draft chapters and prepare the country profiles with a view to presenting the draft report for adoption at the CEPEJ plenary meeting in June 2024.

16.          The Secretariat reiterates the need to stabilise the data used for standardisation (population, GDP) at an earlier stage, and by 1 June at the latest.

17.          The members of the Group agreed to supervise the various chapters of the report while they are being drafted by the experts, as had been done over the last two cycles.

e.                Experts in charge of drafting the report

18.          The Secretariat explained the new procedure for recruiting experts, which will be based on a call for tenders to be launched shortly.

f.                 Visibility, use and dissemination of the report

19.          The Working Group recalled the need to improve visibility and discussed the use and dissemination of the report. The evaluation questionnaire of the report presented by the Secretariat mentions various good practices that could be implemented (e.g., translation of the report, involvement of national training institutions, more systematic training of those responsible for court management in statistics and their uses, the possibility of writing contributions for national legal reviews, the drafting of press releases and use of video clips to promote the report).

20.          The Group also proposes that CEPEJ-STAT training be more systematically integrated into CEPEJ cooperation programmes.

g.                Use of data collected by other Council of Europe departments and request for access to data

21.          The Secretariat informed the Working Group of the increasing number of requests from other Council of Europe departments to use the data collected by the CEPEJ (departments for the execution of judgments of the ECHR, gender equality or children's rights).

III.              SPECIFIC STUDIES ON JUSTICE PROFESSIONALS (2020 DATA)

22.          The Secretariat informs the Working Group of the publication on the CEPEJ website of the two specific studies on judicial experts and on enforcement agents, prepared respectively by the European Expertise and Expert Institute (EEEI) and the International Union of Judicial Officers (UIHJ), which had been adopted by written procedure in September 2023.

23.          The representatives of the EEEI and the UIHJ are pleased with the adoption of the studies and the co-operation with the CEPEJ in this framework.

24.          The specific study on notaries, prepared by the Council of the Notariats of the European Union (CNUE), will be finalised very shortly and sent to the Working Group for adoption by written procedure.

IV.              PREPARATION FOR THE 17^th MEETING OF CEPEJ NATIONAL CORRESPONDENTS (18 OCTOBER)

25.          The meeting of CEPEJ national correspondents on 18 October is being organised in the form of workshops, as last year, but this time throughout the day.

 

26.          The various themes organised into several sessions which national correspondents can attend according to their preferences (see meeting agenda, Annex III) are as follows: the role of the new national correspondent, ICT issues, data collection, the use of CEPEJ-STAT and CEPEJ-COLLECT and the use and visibility of the evaluation report.

27.          These workshops will be an opportunity for the national correspondents to discuss their roles, experiences, good practices and difficulties during the evaluation cycle with the members of the Working Group and the Secretariat.

28.          The Secretariat will also be available to CEPEJ national correspondents throughout the day to discuss ongoing data collection and data quality control for each country.

29.          Annex IV to this document summarises the discussions held at these workshops.

V.               COOPERATION WITH THE EUROPEAN COMMISSION

a.                Study for the European Commission on the functioning of judicial systems in the Member States of the European Union (Justice Scoreboard)

30.          The next CEPEJ study for the EU Justice Scoreboard, which will be based on 2022data, is currently being prepared.

 

31.          The first part, consisting of comparative tables and comments, will be sent to the European Commission on 10 December. The updated country profiles and tables will be sent to the European Commission on 15 January.

b.               Cooperation with the European Commission/Eurostat

32.          The Secretariat informed the Working Group of the ongoing negotiations with the European Commission with a view to the transmission of certain data collected by the CEPEJ in the framework of the EU Justice Scoreboard (relating to case-flow) for publication by Eurostat.

33.          A proposal to amend the technical conditions of the existing contract is being prepared and will be submitted to the Committee of Ministers of the Council of Europe in November.

c.                Dashboard Western Balkans

34.          The Secretariat presented the latest report sent to the European Commission to the Working Group, including a number of improvements to the training indicator and the analysis of beneficiaries profiles for this cycle.

35.          The next data collection (reference year 2023) will open at the end of October.

36.          A mission was organised to Kosovo[*] in October 2023, during which a general presentation of data collection and bilateral meetings with all those involved in data collection took place.

37.          A next mission to Montenegro, which has expressed its interest in a similar mission, could be organised in 2024.

38.          In addition, a regional workshop will be held in Budapest in November 2023 to discuss common issues and best practices in the management of case-flow and the efficiency of procedures.

d.               Dashboard Eastern Partnership countries

39.          The next collection of data (reference year 2023) will open at the end of October 2023. The next data collection cycle will be a transitional year for this project insofar as, at the request of the European Commission, the next report will be based on the most recent data, along the lines of the Dashboard Western Balkans. The data collected will therefore change from the reference year N-2, as was the case for the first two cycles, to the year N-1: the 2023 data will therefore be presented in the report sent in 2024. The data collected as part of the CEPEJ biannual evaluation cycle (reference year 2022) will also be included in this report, for the questions common to the biannual cycle and the Dashboard Eastern Partnership.

40.          A mission to support data collection was organised in Chisinau (Republic of Moldova) in September 2023.

41.          The Secretariat stresses that there are more and more synergies between the two dashboards, particularly as regards the questionnaire, which is now identical, and the content of the reports.

VI.              DISCUSSION ON THE FUTURE CEPEJ-GT-EVAL TERMS OF REFERENCE (2024-2025)

42.          The Working Group held an exchange of views on the tasks that could be included in its future Terms of reference for the period 2024-2025. It was suggested that the role and training of national correspondents should be strengthened, and that the visibility of the report should be improved.

VII.            TRAINING ON CEPEJ EVALUATION TOOLS - HELP MODULE

43.          The Secretariat presented the state of progress and the current format of the CEPEJ-GT-EVAL module.

44.          The aim is to present the HELP training module on CEPEJ tools at the next plenary meeting in December 2023 and to launch the course at the end of the year.

VIII.            ANY OTHER BUSINESS

45.          The Secretariat mentioned a proposal from the CEPEJ Quality Working Group (CEPEJ-GT-QUAL) to improve data collection at national level on mediation. If this proposal materialises in the next mandate of the CEPEJ-GT-QUAL, it will be carried out in coordination with the CEPEJ-GT-EVAL.

Annex I / Annexe I

Annex II / Annexe II

List of participants / Liste de participants

MEMBERS / MEMBRES

PRESIDENT OF THE CEPEJ/ PRESIDENT DE LA CEPEJ	Francesco Depasquale The Honourable Mr Justice, Superior Courts, VALLETTA
PRESIDENT OF CEPEJ-GT-EVAL / PRESIDENT DU CEPEJ-GT-EVAL	Jaša VRABEC Head of the Office for Court Management Development, Supreme Court, SLOVENIA
MEMBER OF THE CEPEJ-GT-EVAL / MEMBRE DU CEPEJ-GT-EVAL	Joanne BATTISTINO Ministry of Justice and Home Affairs, The Law Courts, MALTA
MEMBER OF THE CEPEJ-GT-EVAL / MEMBRE DU CEPEJ-GT-EVAL	Christophe KOLLER Researcher, Director ESEHA, Centre for counselling and comparative analysis, SWITZERLAND
MEMBER OF THE CEPEJ-GT-EVAL / MEMBRE DU CEPEJ-GT-EVAL	Simone KRESS Judge, Vice-President of the District Court of Cologne, Germany
DEPUTY MEMBER OF THE CEPEJ-GT-EVAL / MEMBRE DU CEPEJ-GT-EVAL	Seçkin KOÇER Judge, Directorate General for Strategy Development, Ministry of Justice, ANKARA

PARTICIPANTS / PARTICIPANTS

EUROPEAN EXPERTISE AND EXPERT INSTITUTE / INSTITUT EUROPEEN DE L'EXPERTISE ET DE L'EXPERT (EEEI)	Benoit de CLERCK Co-Chairman of the EEEI
INTERNATIONAL UNION OF JUDICIAL OFFICERS / UNION INTERNATIONALE DES HUISSIERS DE JUSTICE (UIHJ)	Patrick GIELEN Secretary of the UIHJ

SECRETARIAT / SECRETARY

Muriel DECOT Executive Secretary of the CEPEJ / Secrétaire exécutive de la CEPEJ

Christel SCHURRER Secretary of the CEPEJ-GT-EVAL / Secrétaire du CEPEJ-GT-EVAL

Lidija NAUMOVSKA Statistic coordinator/ Coordinatrice des statistiques

Milan NIKOLIC Administrator / Administrateur

Guergana LAZAROVA-DECHAUX Administrator / / Administratrice

Tony VILELA Statistician (Dashboard Eastern Partnership) / Statisticien (Tableau de bord du partenariat oriental)

Federica VIAPIANA Administrator, Project manager (Dashboard Western Balkans) / Administratrices, Chef de projet (Tableau de bord des Balkans Occidentaux)

Daniela ZOL Statistician (Dashboard Western Balkans) / Statisticienne (Tableau de bord des Balkans Occidentaux)

Domitille GAUTIER DE CHARNACE Project manager (Dashboard Eastern Partnership) / Chef de projet (Tableau de bord du partenariat oriental)

Oxana GUTU Project manager (Dashboard Eastern Partnership) / Chef de projet (Tableau de bord du partenariat oriental)

Sabina STANCU Legal officer / Juriste

Enes SEHIC Project manager (Dashboard Western Balkans) / Chef de projet (Tableau de bord des Balkans Occidentaux)

Sandrine LUTZ CEPEJ assistant

Giovanni TOTOZAFY Magistrate at the Department of Continuing Education, E-learning and Partnership of the National School of Magistrates and Registrars of Madagascar. Training at the INSP

Annex III /Annexe III

Agenda of the 17^th meeting of CEPEJ national correspondents

13.00-14.30   Lunch break

14.30-16.00   Third workshop session (Room Bar des Parlementaires)

                                 You can choose to attend one of the following workshops:

Workshop “Quality check” (permanent during the whole day)

Ongoing quality control on national data: individual meetings with the Secretariat in charge of quality control for each country

        Workshop “Data collection”

        Exchanges of experiences on internal data collection

-     What are the main difficulties with the data collection?

-     What tools could help you?

-     Are there any good practices for data collection?

        Workshop “CEPEJ-STAT”

        Suggestions and feedback for a better use of CEPEJ-STAT and CEPEJ-COLLECT

-     How to improve the use of CEPEJ-STAT and CEPEJ- COLLECT? 

-     What are the difficulties of using these tools? 

-     How to make these tools more intuitive and easier to use?

Workshop “Report”

Brainstorming on the use and dissemination of the report

-       Once published in the CEPEJ-STAT report, are the data used in your country?

-       How can we promote our joint work and the CEPEJ report?

Workshop “ICT questions”

16.30-17.00   Coffee break

17.00-17.30   Conclusions (Room 2)

Annexe IV / Annexe IV

17th CEPEJ National Correspondents meeting - Summary of the Workshops

Document prepared by the CEPEJ Secretariat (Document CEPEJ-GT-EVAL(2023)12)

1.     Workshop “New national correspondent”

CEPEJ-GT-EVAL members and Secretariat welcomed the newly appointed CEPEJ National correspondents presenting :

·         the different on-going data collection (Biannual Evaluation cycle, CEPEJ Study for the EU Justice Scoreboard, Dashboard Western Balkans and Dashboard eastern Partnership countries);

·         the different tools available (questionnaire, explanatory note, FAQ; excel file with previous data) and possible supports provided by CEPEJ-GT-EVAL members and Secretariat (visio/ad hoc visit in the country/peer review);

·         importance and use of the data and comments;

·         the different calendars and every steps of these calendars;

·         explanations on the steps of the process (data collection/quality check/preparation of the reports namely)

·         methodological elements namely about comparability of data

·         principles guiding the process (transparency during the full process)

·         role and tasks of the national correspondents.

CEPEJ Secretariat will prepare a “welcome package for the national Correspondent” including these different elements

 

2.     Workshop “Data collection”

The workshop aimed to facilitate brainstorming sessions with the national correspondents and exchange and share experiences on internal data collection.

a.     Data collection process

Preparation of the process

·         National correspondent should identify all the institutions relevant for answering CEPEJ questions during the data entry and quality check process and divide the questionnaire and explanatory notes accordingly so that each institution receives questions for which they are responsible and corresponding explanatory notes.

·         It is recommended to share the explanatory note version with track changes. The indicated track changes reflect the modifications and the latest interpretations. It should be noted that even the questions for which the reply seems to be obvious can change or their interpretation can evolve. National correspondents should be always aware of possible updates in explanatory notes and importance of continuous consulting the content of the Explanatory notes.

·         It is recommended to explain to the institutions that they should provide replies in accordance with the CEPEJ definitions contained in the CEPEJ Explanatory Note. In case of a doubt or conflict between definitions used in a national legislation and CEPEJ definitions, the advantage should be given to CEPEJ definitions in order to ensure comparability of data with other states/entities. 

·         National correspondents could prepare a “welcome package” for contact persons in institutions involved in data collection that would contain general information about CEPEJ evaluation process, links to the relevant CEPEJ website pages, timeline of data collection, explanation of the quality check process, and checklist with the important instructions when providing data (such as read carefully explanatory note, consult previous reply, provide general and specific comments, address discrepancies etc).

·         Sharing previous replies(validated by the CEPEJ) with beneficiary institutions are of utmost importance. National correspondents should highlight the importance of providing consistent replies and providing explanations in each case when a reply has changed. In this way the proper information is shared, the process of collecting replies is accelerated and possibility of mistakes and inconsistencies is minimized. The previous replies are available in PDF format (in the CEPEJ COLLECT and on the following page https://www.coe.int/en/web/cepej/country-profiles ), but also in the excel format shared by the CEPEJ Secretariat at the beginning of every cycle. Nota bene: institutions that provide the data should not rely on the replies they originally sent in previous cycle, but on the final replies/comments that were accepted and validated by the CEPEJ at the end of the quality check process.

·         An identified good practice for questions where replies are likely to remain, is to send the previous replies to institutions, and just ask to “review and confirm the previous reply”. 

·         It is recommended to explain to the institutions the significance of general and specific comments for proper interpretation of the replies provided. It should be highlighted that all comments will be published and will be publicly available in the online database CEPEJ STAT (available at https://www.coe.int/en/web/cepej/cepej-stat). This might contribute to the increase of number and quality of comments. It is to be noted that the comments should be short (up to half page), precise and not containing the “quotes” from the law, but they have to be explained in a plain language.

·         A good practice is to ask institutions involved in data collection to explain in the comments the structure of data provided in the replies (what national data are included in the reply and what is excluded, e.g. what types of cases are included in “other cases”). Providing this information helps better understanding of the replies and ensures consistency as it serves as guidance for providing replies in the future.

·         Furthermore, national correspondent should explain that any discrepancy in the replies with previous cycle should be explained. For quantitative data, generally the focus should be on discrepancies higher than 20% but this also depends on the nature of the question. For example, some questions would require an explanation for much smaller discrepancies (e.g. population, GDP per capita) whereas some larger percentual discrepancies should not be addressed at all (e.g. number of disciplinary proceedings increased from 1 to 2 doesn’t have to be explained, because it is only a one case difference). In the later example, if the CEPEJ COLLECT system indicates discrepancy and requires a comment, it should be explained that reply relates to small absolute values.

·         National correspondents should define deadlines for collection of replies from different institutions and ensure strict control of these deadlines. The good practice is to put earlier deadlines in order to allow more time to national correspondent to review/filter data and identify if additional comments and explanations of discrepancies are needed.

·         Itis recognized as a good practice for national correspondents to explain consequences of missing deadlines and inconsistent replying to the institutions involved in data collection by highlighting that replies might be replaced with NA (not available). Some national correspondents indicate in the comment that an institution failed to provide reply. The respective institution should be informed in advance that such note will be included in the CEPEJ questionnaire.

·         National correspondents should define ways of communication and exchange of documents and information.  The good practice is to organize a meeting at the beginning of the cycle (or set of meetings) with contact persons from institutions in order to explain important points and agree on the process.

·         If there are sufficient resources, the reported good practice is to develop an IT tool (online application) for the collection of replies fromdifferent institutions.

·         The institutions involved in data collection should be familiar with the CEPEJ quality control process. They should be informed in advance that some additional questions regarding replies might be sent by the CEPEJ secretariat. These additional questions usually request additional data or explanation of discrepancies. They have to be addressed by the institutions involved in data collection or replies might be replaced with NA (not available) in accordance with the CEPEJ methodology.  With the full information shared in advance, the institutions will be more prepared to promptly react and provide required updates timely during the quality check process.

Support from CEPEJ Secretariat

·         Communication with the CEPEJ Secretariatduring the data collection is essential and encouraged, as it provides a possibility to timely inform the Secretariat about any difficulties encountered during data collection, ask for guidance and assistance. 

National correspondents should use the capacities of CEPEJ Secretariat in sense of advisory and technical support during the entire process of data collection. Good practices include organizing videocalls, as well as specific support and peer reviews in the country.  These activities are opportunity to provide more explanations of CEPEJ questions and definitions, understand better national legislation and available data, define which national categories should be included in or excluded from the replies, what discrepancies should be explained etc.  They gave good results in practice as they ensured fast and direct exchanges of views and facilitated agreeing on the way forward.

Ø  Use of video calls is recommended when there are questions relating to a specific section of the questionnaire that needs to be fast resolved.  These meetings are brief and focused on a limited number of issues. They are usually organized with national correspondent and/or selected representatives of an institution in charge for replying the section under discussion. 

Ø  Specific support and peer reviews are similar way of assistance provided by the CEPEJ Secretariat to national correspondents. They consist of organizing a one or two-day visit to the member state/entity where representatives of the CEPEJ Secretariat meet with national correspondent and the network of contact persons from different institutions in a set of meetings. Peer review missions include participation of the CEPEJ experts as well. Both specific support and peer reviews are usually organized when there are several sections in the questionnaire that have had outstanding issues which need to be addressed. This is also an opportunity to support national correspondent in their work, provide general overview of CEPEJ work and strengthen the network of contact persons. Previous experiences confirm that these visits increase interest and motivation in the national correspondents’ network. 

During the workshop some other ideas were discussed such as:

·         To create a simplified conversion table to send to the NCs once the new campaign is launched (this table should include a column where we say whether a question is new or has been amended compared to the previous cycle). CEPEJ Secretariat will investigate the way to provide such a table

·         To have a global Timeline of activities (including when to ask data from national sources, enter data, QC, publication, promotion) CEPEJ Secretariat will prepare a “welcome package for the national Correspondent” including these different elements

Entering the replies in the CEPEJ COLLECT and quality check

·         Before sharing the reply with CEPEJ, and if possible, one good practice could be for national correspondents to organize an “internal quality control of data”. This step includes verifying the collected replies to determine if there are no obvious omissions and errors (such as typos, unfinished comments, errors in summing numbers, reply submitted doesn’t correspond to the question, and similar). Furthermore, it is advisable for national correspondents to compare collected replies with the definitions in the explanatory note, as well as with replies from previous cycle. It is also helpful if national correspondents check what issues were raised during quality check in the previous cycle and verify if they have been addressed in the new replies. This sort of internal quality control will reduce questions form the CEPEJ Secretariat and consequently shorten the time of the quality check.

·         It is recommended that institutions involved in data collection are invited and trained to conduct these checks themselves before providing the replies to the national correspondent.

·         Once the data are entered in the COLLECT, the system will also indicate certain discrepancies and automatically ask for explanations in the “discrepancy comment boxes” under the question. These discrepancy comments will now be shown even when replies are saved as pending (recent modification of the CEPEJ COLLECT). These comments should be provided before the replies are finally posted.

·         Progressive entering of repliesin the COLLECT is recognized as a good practice. The national correspondents do not have to wait for all replies to be collected and can enter the replies as they receive them from different institutions. This practice allows more time for requiring missing data or comments and might optimize the data quality check procedure. It should be noted that only the data which are final and not expected to be changed should be posted.

·         Due to technical limitations of the system, it is recommended to save replies often (option “save in pending”) to prevent slow upload and potential loss of entered replies. 

·         Finalizing all entries early, in months before deadline (e.g. July and August), is also recognized as good practice as it gives possibility for earlier quality check and more time to provide additional explanations when needed.

·         When the first round of quality check is finished, the CEPEJ Secretariat will send an information about it via email. National correspondents should ensure quick distribution of CEPEJ additional questions to the contact persons. They should define internal deadlines for responding.

·         If an institution refuses to revisit/change the dataduring the quality check upon national correspondent’s request, the reported good practice is to try mediation by a third party (representative of another department/institution). A meeting (video call) with CEPEJ Secretariat is also an option in order to understand better the issue and define the way forward (see section above).

Follow-up

·         At the end of this process, it is a good practice to make/update a list of all institutions which will contain the name of the institution, contact details and all questions for which each institution is responsible for. This list might be helpful in next data collections and it ensures consistency in case of a change of the national correspondent.

·         In order to facilitate data collections in the future and ensure consistency of replies, building institutional memory is of utmost importance. Work of the national correspondents should be organized in a way which ensures continuity of the activities and consistency of replies without jeopardizing process in case of changing national correspondent and/or contact persons in different institutions involved in data collection. Different good practices were presented:

o    Maintain a  record/list of national sources (institutions/individuals) and data that were used to provide replies validated by the CEPEJ.

o    Explanation of which national data were included in the reply of each question should be as specific and precise as possible.

o    national correspondents and their network of contact persons could consider creating a manual to be updated at the end of each cycle that will contain all CEPEJ questions with references to institutions, national sources and national data used to reply CEPEJ questions. This manual would have to be updated after each cycle. 

o    Raising awareness and increasing the knowledge and capacities of the institutions in national correspondent’s network to reduce time of data collection and quality check.

·         Once the CEPEJ report is published, national correspondents are invited to share the CEPEJ report and media articles with all institutions involved in data collection. A good practice is to organize a meeting to present report, country profiles, CEPEJ STAT, and media reports.  This helps increase visibility but also understanding of the purpose of the whole data collection process. It could also help increase the motivation in the national correspondent’s network and could contribute to more efficient data collection in future.

b.    Workshop “Report”

The workshop aimed to facilitate brainstorming sessions with the national correspondents and generate ideas to further promote the dissemination of the CEPEJ Evaluation report.

There was a consensus that the promotion of CEPEJ data should be intensified. The following ideas for promoting the report and the data collected by CEPEJ were put forward:

·         Compilation of a checklist outlining the duties and functions of National Correspondents (NC) throughout the entire process, including their responsibilities during the promotion of CEPEJ report. This measure aims to provide clear guidance and recognition to the NCs, giving them mandate delegated from CEPEJ and recognized by their institution to pursue their list of duties and ensuring their active involvement and accountability including the effective promotion and dissemination of CEPEJ report.

·         Arrangement of yearly hands-on training sessions on CEPEJ STAT to bolster the proficiency of users and their understanding of data analysis. Incorporating training modules on CEPEJ STAT into all CEPEJ projects and peer review missions to facilitate the dissemination of knowledge on CEPEJ data.

·         Annual leaflet or newsletter to provide stakeholders with a comprehensive overview of recent CEPEJ tools and events, fostering better engagement and awareness of CEPEJ's initiatives beyond Evaluation activities.

·         Creation of short thematic videos on specific topics that serves the purpose of capturing the interest of targeted stakeholders. These videos can effectively communicate crucial information, making it more accessible and engaging for the intended audience. By focusing on topics relevant to these stakeholders, the videos can deliver key insights and promote a deeper understanding of the data and its implications within their specific domains.

·         Communication plan development: drafting a comprehensive communication plan outlining the scheduled release dates for our upcoming news, tweets, articles, and other content. This plan will facilitate direct sharing with the media departments within the Ministries/Institutions, enabling them to efficiently repost and share the information and ensure ultimately a consistent and coordinated dissemination of relevant updates and publications, effectively amplifying the reach and impact of CEPEJ report.

c.     Workshop “CEPEJ-STAT”

CEPEJ Secretariat presented the application for data entry CEPEJ COLLECT and the interactive database CEPEJ STAT in this workshop.  Main features of both tools were presented followed by discussion on the problems and suggestions for improvement and or promotion. The list below summarises the comments discussed during the two sessions of this workshop.

CEPEJ-STAT

·         Promotion

o    All participants agreed that CEPEJ-STAT should be more promoted using different medias. CEPEJ Secretariat will make a proposal on how its visibility and use could be enhanced  and what might be the role of NC for this

o    The participants also considered that making CEPEJ STAT more user friendly by improving the navigation to different dashboards will make it more accessible. CEPEJ will investigate the way to improve the user friendliness of the dashboards, provide more training to popularise and make the dashboards easier for users.

o    To ease the use of CEPEJ STAT the NC considered that Manual for its use will be helpful. New manual is already under preparation by the Secretariat.

·         Content and User Interface

o    The participants agreed that a short description for each dashboard and explorer is important to understand its main purpose and in this way the user will know which dashboard is relevant for their need.  CEPEJ Secretariat will add this feature on the main page of CEPEJ STAT.

o    Create some new dashboards. For example, allowing different questions comparison for analysis. The Evaluation working group of the CEPEJ (CEPEJ-GT-EVAL) every cycle reviews the current dashboards/explorers and present a new one and or retire existing. Same applies to the next publication NC are encouraged to propose dashboard that they think will best suit their needs.

·         Better navigation

o    Home button (on Tableau public profile of CEPEJ) bringing back to CEPEJ-STAT CoE page. The link exists and CEPEJ Secretariat will try to put forward this access.

CEPEJ-COLLECT

National correspondent agreed that there are some difficulties with the reaction time or/and disconnection timeout of the CEPEJ-COLLECT application. The secretariat is fully aware of this issue, and it is trying to solve the problem with the IT department.

There were also number of other proposals how to improve the application technically, its content or the user interface. CEPEJ Secretariat is analysing all the proposals and all that do not require structural change and are considered relevant for all users will be considered for evolution of the system. 

In each workshop there were also ideas that overlapped with other workshop or they were of general nature. Few examples are for example a personal note system for the NC to write information to be checked and remembered by them 2 years later; User satisfaction survey and/or external evaluation of the system/s.

d.    Workshop “ICT questions”

The workshop was more to inform the National Correspondent on:

·         why the questionnaire was changed and what was changed;

·         the changes made in the ICT part of the questionnaire;

·         Explanation on the newly introduced usage rate;

·         Emphasis on the importance of comments since it is first cycle of the new ICT questionnaire;

·         Emphasis on reading the Explanatory notes where an example on how to calculate the deployment and usage rate .

The general feedback was:

·         ICT questionnaire is clearer and easier to fulfil than previous cycles (the feedback given by the ICT sources of the NC within their country)

·         still, some countries have problem to get answer from their ICT departments because of lack of interest.

·         NC encountered difficulties to estimate usage rate or/and deployment rate. The advice was to first check in the examples given in Explanatory Notes but also to add comments that will help CEPEJ to advise if the selected answers are valid or how to calculate/estimate the reply.  Whenever the situation is specific the case could be discussed directly with CEPEJ.

·         The focus of the ICT questions is on litigious questions (civil = civil and commercial litigious and the same for criminal and administrative). The issue was raised by one NC and the answer is public in Q&A