Strasbourg, 18 March 2021                                                       

                                                                                                            CEPEJ-GT-EVAL(2019)2rev1 

       

EUROPEAN COMMISSION FOR THE EFFICIENCY OF JUSTICE

 (CEPEJ)    

 

WORKING GROUP ON THE EVALUATION OF JUDICIAL SYSTEMS

(CEPEJ-GT-EVAL)

Table of Contents

1     Introduction. 2

2     Data entry in CEPEJ-COLLECT. 3

3     Quality control 4

3.1          Distinction NA/NAP.. 4

3.2          Quality check – quantitative data. 5

3.2.1              Variations. 5

3.2.2              Consistency. 6

3.3          Quality check – qualitative data. 12

4     Comments: general and specific. 14

1             Introduction

Purpose of the document is to provide an overview to the methodology of the quality control that is essential phase of the CEPEJ evaluation process. This document should be consulted by the National Correspondents together with the Explanatory Note where the purpose and clarifications of the questions from the CEPEJ Evaluation Scheme are provided, as well as the CEPEJ-COLLECT user manual that gives the overview on the use of the CEPEJ data collection tool.

The data quality control process (“quality check”) is an essential stage of the evaluation exercises carried out within the framework of the CEPEJ Evaluation Working group (CEPEJ-GT-EVAL). This is a phase of cooperation between the national correspondents (NC) and the CEPEJ Secretariat (Secretariat) aimed at ensuring a sufficient quality and consistency of the data appearing in the Evaluation Report on European Judicial Systems of the CEPEJ, in CEPEJ-STAT as well as in the other studies of the CEPEJ based on the collected data.

The data quality control process is integral part of the CEPEJ-COLLECT system and the communication between the Secretariat and the national correspondent takes place within the system. Indeed, even if the primary function of the CEPEJ-COLLECT system consists of collecting all the necessary information/data, it also includes a platform for regular exchanges between the national correspondents and the CEPEJ Secretariat for the purpose of data collection and quality control of these data.  

The process of data collection includes several steps before data can be considered final. First, the national correspondent fills in the requested information/data (quantitative or qualitative) and posts the completed sections.

In the second step, the Secretariat ensures that data is consistent with the previous cycles, but also in line with the CEPEJ’s own methodology for presentation of data (for example, vertical/horizontal consistency; use of NA/NAP answers etc.). Besides, the Secretariat ensures the consistency of the data provided by all participating States, which makes it possible to present comparable data at the end of the process.  Within this step, the quality control is carried out as an iterative process of questions and answers exchanged between NC and Secretariat.

While some variations in data between different cycles can naturally be observed, they will have to be explained within the frame of the “quality check” process so that developments in judicial systems can be best interpreted in the most accurate way when they are used in reports, studies or on CEPEJ-STAT. The clarifications requested by the Secretariat during the data quality control process often improve the comprehensibility of the data, but also the understanding of the basic principles and particularities of each national system. Accordingly, the comments provided by the national correspondents in order to explain the data and their contexts, and eventually the variations noticed between different cycles, are of considerable importance. Indeed, the CEPEJ methodology insists on the fact that all data should be read and analysed jointly with the comments provided by the national correspondents.  

Furthermore, in accordance with the methodology developed by the CEPEJ, the national correspondents are responsible for the quality of the data provided. The role of the Secretariat is to assist the NC in the process by drawing the attention to potentially incorrect information and invite NC to provide explanations. Therefore, the Secretariat, cannot modify any data without discussing it with the national correspondent first. Nevertheless, the Secretariat may warn the national correspondent that a particular data will be replaced by “NA” if the data or the comment it is not deemed to be of sufficient quality.  

The purpose of this document which has been prepared by the CEPEJ Secretariat is to explain the way this quality control is carried out within the Secretariat, under the supervision of the CEPEJ Evaluation Working Group.

2             Data entry in CEPEJ-COLLECT

The process starts with the initial data entry of answers to questions that are organized in sections.

To post/publish (finalise) a section, the national correspondent must answer all the questions contained in it. If this is not the case and one or more questions have not been replied, the section cannot be published. Nevertheless, section can be saved without publishing (“Save in pending”) for protecting the data entry and continuing further.

The significant variation between the replies (quantitative data) in the current data collection and those of the previous cycle is automatically detected by the CEPEJ-COLLECT system and reported to the user. In that case the section can only be posted if the national correspondent provides an explanation in the form of a comment, or modifies (in case of mistake), the data entered. 

3             Quality control

For each section posted (and therefore completed) by the national correspondent, the Secretariat will check the data communicated and, if necessary, ask questions and request clarifications through the yellow heading at the bottom of the page "Communicate to user".

In turn, the national correspondent can insert his/her replies by pressing “Communicate to user”. This process finishes when the data in the section is cleared.

3.1         Distinction NA/NAP

The reply NA (non-available) indicates that the category in question or the situation described exists in the concerned judicial system, but that the data/information is not available (either for the current cycle or in general). On the other hand, the reply NAP (not applicable) means that the category or situation in question does not exist in the judicial system, which makes the question irrelevant.

Examples[1]:

The reply NA informs us that in the concerned judicial system there are “legal advisors who cannot represent their clients in court”, but that their exact number is not available.

The reply NAP informs us that in the concerned judicial system the category of “legal advisors who cannot represent their clients in court” does not exist.

3.2         Quality check – quantitative data

3.2.1      Variations

The CEPEJ-COLLECT data collection system automatically identifies variations between the quantitative data that the national correspondent is providing for the current cycle and the data that had been validated for the previous exercise. Put differently, the system will alert the user/national correspondent if the variations are too high (% of variation defined by the Secretariat under the supervision of the CEPEJ Evaluation Working Group – CEPEJ-GT-EVAL) compared to the previous cycle. In this case, and in order to be able to save the data, the national correspondent will have to either modify the data (in case of a mistake) or justify the difference in the comment that appears in the red box.

The system does not allow the national correspondents to publish a section until the relevant data is explained or modified.

Example 1:

In this case, the alert message indicates the existence of a variation (red box) which requires explanation. The national correspondent has to provide an explanation preferably in the comment box below the question and indicate this explanation in the red comment box. After completing these steps, the national correspondent will be  able to publish the section.

Example 2:

In this example, the alert system had noticed important variations for all three categories (red colour). The provided comment explains the observed variations by a reform of the judicial map that has been carried out since the previous cycle.  

3.2.2      Consistency

As a general rule, for quantitative data organized in a table, the total should correspond to the sum of the subcategories (presented horizontally or vertically). The latter are indeed components of the total. For example, the total number of judges (Q46)/prosecutors (Q55)/lawyers (Q146)/notaries (Q192) etc. must correspond to the sum of the number of women and the number of men. Besides, the total number of judges (Q46) must correspond to the sum of judges sitting at all instances (from the first to the last). Similarly, the budget allocated to the functioning of all courts (Q6) must correspond to the sum of all existing budgetary components.

This general rule may have exceptions provided they are explained. Furthermore, the replies “non-available” (NA) can affect the application rules of these principles.

a.   Vertical consistency

Principle

The subcategories appear below the total. They are always numbered, and the total should equal their sum (Total = 1+2+3+4+5).

Example 1:

The total equals the sum of the four subcategories (26 = 5+20+1+0).

 

Example 2:

The total number of criminal cases for each column (pending cases on 1 January or 31 December, incoming cases, resolved cases) corresponds to the sum of the two subcategories (severe criminal cases and misdemeanour and/or minor criminal cases) (218 = 206 + 12; 656 = 545 + 111; 662 = 552 + 110; 212 = 199 + 13).

Exceptions:

·         If one of the subcategories is NA (non-available), the total cannot be equal to the sum of the other subcategories for which the replies are quantitative data. More precisely, the total will necessarily be NA.  

Example 1 – the reply NA is provided in respect of one of the subcategories

	Approved budget (in €)	Implemented budget (in €)
TOTAL - Annual public budget allocated to the functioning of all courts (1 + 2 + 3 + 4 + 5 + 6 + 7)	NA	NA
1. Annual public budget allocated to (gross) salaries	1000	1000
2. Annual public budget allocated to computerisation	NA	NA
3. Annual public budget allocated to justice expenses (expertise, interpretation, etc.)	1000	1000
4. Annual public budget allocated to court buildings (maintenance, operating costs)	2000	2000
5. Annual public budget allocated to investments in new (court) buildings	5000	5000
6. Annual public budget allocated to training	2000	2000
7. Other (please specify)	1000	1000

In this example, the subcategory 2. “Annual public budget allocated to computerisation” exists in the judicial system in question and constitutes a component of the budget allocated to the functioning of all courts. However, the precise amount of this budgetary component cannot be identified. Therefore, the total cannot exclude this subcategory and correspond to the sum of the subcategories for which data is available (total ≠ 1+3+4+5+6+7). Thus, the NA reply for one of the subcategories requires an NA reply for the total. 

N.B. Opposite to NA, if the reply for one subcategory is NAP meaning that this category does not exist and there is no budget allocated to this category, the total can be calculated on the basis of the figures provided for the remaining subcategories.

N.B. For very specific situations different from the examples listed above, explanations should be added in the comments and solution will be agreed with the Secretariat.

·         if several replies are NA, the total can be either a value (which will necessarily be higher than the sum of the available subcategories) or NA.

Example 2 – despite the reply NA for several subcategories, the total is a value (higher than the sum of the subcategories)  

	Approved budget (in €)	Implemented budget (in €)
TOTAL - Annual public budget allocated to the functioning of all courts (1 + 2 + 3 + 4 + 5 + 6 + 7)	10 000	10 000
1. Annual public budget allocated to (gross) salaries	3000	3000
2. Annual public budget allocated to computerisation	1000	1000
3. Annual public budget allocated to justice expenses (expertise, interpretation, etc.)	NA	NA
4. Annual public budget allocated to court buildings (maintenance, operating costs)	2000	2000
5. Annual public budget allocated to investments in new (court) buildings	1000	1000
6. Annual public budget allocated to training	1000	1000
7. Other (please specify)	NA	NA

In this example, data is not available for two of the seven subcategories (3 and 7). The total represents a value (10000 €) higher than the sum of the subcategories for which the information is available (1+2+4+5+6 = 8000 €). This configuration implies that the extra 2000 € contained in the total are distributed between “justice expenses” (3) and “other” costs (7) but that it is impossible to identify precisely each of these two budgetary components. 

However, the total may be NA if, as per previous example the total of “justice expenses” (3) and “other” costs (7) is also not known.

·         On the other hand, if one or more replies are NAP (not applicable), the total corresponds to the sum of the subcategories, because categories replied by NAP do not exist in the judicial system in question and do not affect the total.  

Example 3 – one or several subcategories are replied by NAP

	Approved budget (in €)	Implemented budget (in €)
TOTAL - Annual public budget allocated to the functioning of all courts (1 + 2 + 3 + 4 + 5 + 6 + 7)	8000	8000
1. Annual public budget allocated to (gross) salaries	1000	1000
2. Annual public budget allocated to computerisation	1000	1000
3. Annual public budget allocated to justice expenses (expertise, interpretation, etc.),	1000	1000
4. Annual public budget allocated to court buildings (maintenance, operating costs)	2000	2000
5. Annual public budget allocated to investments in new (court) buildings	NAP	NAP
6. Annual public budget allocated to training	2000	2000
7. Other (please specify)	1000	1000

In this example, the budget allocated to the functioning of all courts does not include a component dedicated to investments in new court buildings. Therefore, the reply to subcategory n° 5 is NAP (not applicable), while the total is the sum of the rest of the subcategories (1+2+3+4+6+7).

The same applies when the reply is NAP for several subcategories. To the extent that the relevant budgetary components do not exist in the judicial system in question, the total can be calculated despite the presence of NAP replies.

The national correspondent may possibly specify the reason why for certain budgetary elements the reply is NAP (for example, the training may be entirely from the budget of the public institution in charge of the training of judges; the investments in new court buildings may be part of the budget of another ministry etc.).

b.     Horizontal consistency

The subcategories appear horizontally in the total line. There are two possible scenarios:

·         Horizontal consistency in tables with three columns: total, males and females

The total must simply correspond to the sum of men and women

Example:

In the four lines the total is equal to the sum of men and women:

Total (278) = 142 (males) + 136 (females)

First instance professional judges (205) = 104 (males) + 101 (females)

Second instance professional judges (60) = 30 (males) + 30 (females)

Supreme courts professional judges (13) = 8 (males) + 5 (females)

·         The horizontal consistency within questions related to the number of cases

With regard to questions 91, 94, 97, 98, 99, 100 and 101, a special formula applies, namely:

Example:

In this particular case the formula will be presented as follows:

584  + 1987 – 2005 = 566

3.3         Quality check – qualitative data

N.B.: The CEPEJ-COLLECT system does not automatically identify discrepancies in replies with regard to qualitative data (involving a Yes/No reply, optional questions or a discrepancy between NA/NAP). It falls to the national correspondent to carry out these checks:

 

·         Checking discrepancies between cycles regarding Yes/No questions.

As already pointed out, the CEPEJ-COLLECT system does not alert the national correspondent if the reply for the current cycle differs from the reply of the previous cycle when the two possible options are YES or No. However, this verification will be carried out during the data quality control phase. Thus, in case of variation, the national correspondent is expected to provide explanation in the comments on the difference in replies (reform, new interpretation of the question etc.) or correct the replies in case of mistake. 

Example:

During the quality control process, the Secretariat had noted the difference in replies for several categories, but the comment provided by the national correspondent made it possible to validate the data from the first phase of the process.

·         Checking variations between NA/NAP/a data (quantitative or qualitative).

§  NA – the category exists but the data is not available either for the reference year or in general (e.g. no statistics are conducted on the matter, or the national categorization differs from that of the CEPEJ)

§  NAP – the category or the situation in question does not exist in your country (the question is not applicable)

§  0 – the category exists but for the reference year there were no cases referring to it

·         During the data quality control process, a question will be systematically asked to the national correspondent in case of a discrepancy between:

§  NA(previous cycle) and NAP (current cycle) – the category existed before but no longer exists;

§  Number(previous cycle) and NA or NAP (current cycle) – the category existed, and the data was available which is no longer the case;

§  NAP(previous cycle) and quantified data or NA (current cycle) – the category did not exist but was introduced.

The national correspondent is invited to provide a comment at the moment the data entry in case he noticed a discrepancy between previous and current answer;in case for example the NC is aware that in previous cycle the reply was NAP because certain category did not exist and now it is available the comment could already be included before quality control starts. In this way Secretariat will validate the question without posing any questions.

Example:

“(…) Concerning (mandatory) mediation ordered by a judge:

Law 4446/2016 (art. 23), inspired by the 2011/7/EU directive on combating late payment in commercial transactions, provides a new procedure before the administrative courts of appeal using compromise and settlement to resolve a dispute in the field of public procurement”.

The option "ordered by the judge in the context of ongoing litigation" was ticked for the first time for 2016. The comment provided by the national correspondent allowed understanding why for previous cycles the reply was NAP while for 2016 a positive reply is given.

4             Comments: general and specific

·         The general comment is part of a special section within the CEPEJ-COLLECT system.

According to the CEPEJ methodology, the comments are integral part of the CEPEJ evaluation process and the data collected can be interpreted only jointly with the comments. For this reason, the Secretariat pays special attention to the content and the quality of the provided comments.

For each question in the CEPEJ Evaluation scheme, there are two type of comments: General comment and specific comment.

The general comment provides general information that is valid not only for the current cycle, but also for all other cycles. It describes fundamental principles of the judicial system as well as its national specificities. 

The general comment is automatically taken over by the CEPEJ-COLLECT system (from the previous cycle) and automatically appears in the respective window. It is the responsibility of the national correspondent to check whether the general comment is still relevant and to maintain it, or to update it, complete it, reformulate it, make it more concise, etc. If the general comment is no longer valid, it must be removed and finally the relevant new information inserted.

·         The specific comment is to be inserted below the question concerned.

It describes a specific situation concerning the reference year. It can complete the general comment, provide a concrete example in the light of the general comment, or pinpoint a reform or any other important recent evolution.

The specific comment and the general comment will automatically appear in the CEPEJ-STAT system to accompany data. A concise comment gains in clarity and readability. Besides, both comments (specific and general) should not be identical so that the same information does not appear twice in the CEPEJ-STAT system.  

Example:

In this example, the specific comment clarifies the content of the total number of notaries indicated (information valid only for the current cycle), while the general comment provides elements of general information relating to the appointment procedure of notaries, their status etc. (information valid for all cycles as long as the legislative provisions remain in force and are not amended).