2018 International Conference on Big Data and Artificial Intelligence (ICBDAI 2018)

Thoughts and Methods of Applying Artificial Intelligence in Judge Quota

Management
Xin Li1,a, Qingwen Qi1,b, and Xin Tian2,c,*
1
Law School, Sichuan University, Second Section of Chuanda Road, Shuangliu District, Chengdu, Sichuan,
China
2
College of Mathematics, Sichuan University, South Section of Yihuan Road, Wuhou District, Chengdu,
Sichuan, China
a
[email protected], [email protected], [email protected]
*corresponding author

Keywords: artificial intelligence, judicial reform, personnel quota system, judge management
mechanism.

Abstract: Artificial intelligence technology is developing vigorously, but its application in many
fields is still in its infancy. For example, in the judicial field, it mainly concentrates on the trial
work, resulting in conflicts between science and technology and ethics, efficiency and justice.
Therefore, attention should be paid to areas such as trial management where such conflicts do not
arise, such as the judge quota management system, which seeks to objectify and rationalize
management standards. Based on a large number of judgment documents and judicial resources
published at present, through empirical research and interdisciplinary research methods, in this
paper, it is tried to construct a judge quota management mechanism, which focuses on the
construction of multi-source heterogeneous information fusion mechanism, the establishment of
judge business files and case complexity evaluation model, in order to realize the application of
artificial intelligence in the key nodes of judge quota management, and finally provide a new
direction for the current judicial reform.

1. Introduction
With the advent of the era of artificial intelligence (AI), many fields have explored the use of AI
to build a more scientific and efficient production and management model. However, due to the
constraints of practical factors, the application of AI technology in many fields is still in its infancy,
and so is the application in the judicial field.
In the fourth round of judicial reform in China, AI has been innovatively applied to the judicial
system, which is different from the traditional model and injects new vitality into the current
judicial reform. Under the overall framework of the Intelligent Court, AI technology is integrated
into four scenarios: serving the public, serving the trial of cases, serving the execution of judgment
and serving the judicial management [1]. Throughout the current academic research and application
development of AI, the main focus is on trial work, focusing on evidence and sentencing, and some
progress has been made. As a new field of development, judicial AI will inevitably be questioned,
since its application in the trial process is restricted by judicial experience, the autonomy of judges’
discretion [2], the incompleteness of big legal data and the inefficiency of algorithms used by itself
[3], and may lead to conflicts between science and technology and ethics, efficiency and fairness.
Therefore, more attention should be paid to the areas of trial management where such conflicts do
not arise, especially in the quota management of judge.
The existing judge management system based on the administrative system has some problems,
such as too high degree of administration, lack of professionalism and lack of objective standards,
which lead to the slow updating of the team of judges, insufficient incentives, unreasonable
assessment methods and unsatisfactory safeguards. After the implementation of the personnel quota
system, it is more urgent to establish a scientific, objective and real-time response judge
Copyright © (2018) Francis Academic Press, UK 104 DOI: 10.25236/icbdai.2018.017
management system in order to respond to the requirements of the judge quota system for the
objectivity and rationalization of management standards and the fairness of management procedures.
However, relying on the traditional hierarchical management model, it is difficult to achieve the
above objectives. Hence, it is possible to explore the application of AI in the quota management of
judges to set up business files for judges, use big data technology to capture and count the objective
information existing in the case files, form a set of methods and standards for objective evaluation
of judges' work, and apply them to the selection, evaluation, punishment, guarantee and withdrawal
of judges.

2. Methods and Materials

2.1. Research Methods
Two methods will be used in the research. First, empirical research, through on-the-spot
investigation, on-site interviews, data collation and analysis and other methods, a quantitative and
qualitative analysis is made in this paper on the current situation of the quota management of judges
in China and the application of AI in the judicial field, focusing on the tremendous impact of
personnel quota system on the management system of judges and how courts use information
technology to meet new challenges. Secondly, the interdisciplinary research method, which
combines legal knowledge and principles with computer and statistical methods, provides possible
technical solutions for the application of AI in the quota management system of judges. The
development of legal application of AI is not only the trend of judicial practice itself, but also an
example of the extension of AI to all areas of life in society. The field of law provides abundant and
new application scenarios for the study of AI. At the same time, the application of AI also provides
methodological enlightenment and ideological experimental means for the study of law [4].
2.2. Research Materials
In 2016, the reform of the judge personnel quota system was fully launched. In 2017, more than
120,000 judges were born in the people's courts at all levels throughout the country. Through the
implementation of personnel quota system, more outstanding judicial personnel can be selected by
considering the number of cases handled by courts, the population of jurisdictions and the level of
economic development. Personnel classification management optimizes the structure of court
personnel and improves efficiency. It is also more scientific and reasonable to use individual
performance and ability as the primary criteria for selection. However, the implementation of the
personnel quota system may also lead to the phenomenon that the number of cases handled by
judges is more than that before the quota due to the "elitism" of judges and the "de-administration"
of courts which are difficult to promote. At the same time, the more stringent conditions of
enrolment also affect the stability of the trial team. Under the new management mode, the
performance appraisal system has been optimized, but there is still a lack of rationality in the
number of appraisal indicators and the contradiction between performance appraisal itself and the
law of judicial work itself. Quantification of indicators may lead to judges' cheating or even
tampering with data, which will reduce the actual quality of the case.
AI technology is mainly used as a tool to assist judges in handling cases. Through in-depth
analysis of massive judicial data, scientific prediction of referee results is achieved. The most
important basis of the judicial application of AI is the massive judgment documents that have been
published. By November 2018, more than 56 million judgment documents had been published
online in China (see Table 1), an increase of about 97% compared with 1.7 million in the first year
of publication in 2013. The most solid information foundation for AI judicial applications have
been provided by judgment documents, case databases, judicial interpretation databases of laws and
regulations, legal documents databases, periodical databases of legal books, user consultation
databases, open litigation data, audiovisual recordings and videos of court hearings, etc. available at
China Judgment documents Network, China Trial Procedure Information Open Network, China
Executive Information Open Network, National Enterprise Bankruptcy Reorganization Case

105
Information Network, Case Information Disclosure of the People’s Procuratorate of the P.R. China
and other websites and six resource libraries, a total of 10 billion words of case gist, legal point of
view, legal books, legal documents, judicial decisions, legal journals provided by the Legal
Information Platform.
Table 1 Number of cases published by China Judicial Information Network (as of November 2018).
Categories of Judgment Documents Total number
Criminal documents 7,146,692
Civil documents 35,762,237
Administrative documents 1,740,881
Indemnity instruments 46,720
Executive instruments 11,836,664
Total 56,662,433
The relevant quantitative indicators to measure the workload of judges should include the
number of cases received in trial tasks, the number of assistants of judges, the number of trial days,
the number of cases closed, the rate of mediation withdrawal, the rate of judgment, the number of
appeal cases, the rate of retrial, the number of trial tribunals, the area of the jurisdiction of the court,
the financial revenue of the jurisdiction and the per capita income of the jurisdiction. Non-trial tasks
are based on the time when judges participate in meetings, events and business training. As of June
30, 2018, the national court had a total of 124,000 judges, the number of new cases received was
12.295 million, and the number of new cases received per capita was 99.2. According to the
published statistical data of China Legal Yearbook, the relevant statistics of the number of first
instance cases received by the people's courts in recent five years (see Table 2 below) and the
number of first instance cases closed (see Table 3 below) provide a Macro-measurement for
measuring the workload of judges.
Table 2 Statistics on the number of first instance cases received by the people's courts in the past
five years (unit: case/year)
Year 2016 2015 2014 2013 2012
Number of first instance
criminal cases received 1,101,191 1,126,748 1,040,457 971,567 996,611
Number of first instance civil
and commercial cases received 10,762,124 10,097,804 8,307,450 7,781,972 7,316,463
Number of first instance
administrative cases received 225,485 220,398 141,880 123,194 129,583
Total number of cases received
at first instance 12,088,800 11,444,950 9,489,787 8,876,733 8,442,657
Table 3 Relevant statistics on the number of first instance cases closed by the people's courts in the
past five years (unit: case/year)
Year 2016 2015 2014 2013 2012
Number of cases closed in the
first instance of criminal cases 1,115,873 1,099,205 1,023,017 953,976 986,392
Number of cases closed in the
first instance of civil and
commercial cases 10,763,889 9,575,152 8,010,341 7,510,584 7,206,331
Number of cases closed in the 130,964 120,675 128,826
first instance of administrative
cases 225,020 198,772
Total number of cases closed at 12,104,782 10,873,129 9,164,323 8,585,235 832,134
first instance
Note: 1) The first instance cases in Table 2 and Table 3 refer to the cases that the people's courts
106
have jurisdiction over according to the level of litigation and hear according to the procedure of first
instance.
2) Received cases refer to newly received cases in the same year, and the closed cases include
old cases from the previous year.
3) The latest data for 2017 has not yet been released.

3. Current Application of AI in Judge Quota Management

Up to 2016, the large data management and service platform and the launch of "legal
information" platform of the People's Court have met the needs of judges for law, case and
professional knowledge in case trial. The judicial statistics of the national courts are integrated with
the data management and service of the people's courts, so as to realize the automatic generation of
judicial statistics reports. The people's courts have completely bid farewell to the era of artificial
judicial statistics and the inefficient trial management mode.
At present, the pilot application of AI in the quota management of judges in our country has
achieved initial success. With the help of big data and AI technology, many tools have emerged
such as the assistant system of "Judge e Assistant" of Jiangxi Provincial Intelligent Court, which
provides intelligent and integrated auxiliary services for judges in the core trial links of reading
documents, holding court sessions, writing documents and adjudicating, intelligent speech
transcription system with 95% recognition rate in Hulunbuir Intermediate Court, personnel
management intelligent access management equipment of Shanghai Minhang Court and Sichuan
Chongzhou Court. Admittedly, AI has made great achievements in the construction of intelligent
courts at present, but it still needs time to be popularized nationwide. At the same time, the
intelligent system for judge quota management needs to be fully developed and integrated with
other work in the trial.

4. Ideas for Application of AI in Judge Quota Management

The application of AI is based on the judicial big data, which is realized by information
accumulation and mathematical model reasoning. The application of AI in the quota management of
judges should focus on trial assistance, trial management and trial supervision. At present, the
application of AI in key nodes of judge quota management mainly includes the following ideas:
First, by accurately measuring the overall workload of the Academy of Algorithms, the
proportion of specified number of judges is determined.
For a specific court, the determination of the proportion of the enrollment depends mainly on the
two factors of the total annual work of the court and the average annual workload of the judges.
Traditional courts or judges' workload is calculated on a case-by-case basis, which is simple, direct
and easy to overlook the differences in the workload of specific cases. It is possible to use AI to
calculate the total annual work of the court; use the weight analysis method to measure the average
annual reasonable workload of the judge, and finally calculate the proportion of the court quota
through the above two data.
Second, AI is used to test the judge's enrolment.
At present, the selection of judges is mainly conducted after the written examination is passed,
and then interviewed. The interview process generally includes the personal report of the applicant
and the scoring by police officers in the courtroom where the applicant works according to the
working performance, competence and results, as well as by leaders, management office and
disciplinary inspection room on the basis of thought and conduct, working competence, working
attitude, performance and integrity. Subjective statement and artificial grading can hardly reflect the
judge's objective situation and real trial ability, but at the same time, they are more painstaking.
Therefore, adding VR technology in the trial ability test link with the immersion and interaction of
virtual reality enables applicants to devote themselves wholeheartedly to the simulated test
environment to show the actual trial ability.
Thirdly, AI is used to assign jobs to specified number of judges.
107
 Based on the similarity of case procedure and the particularity of case cause, the type and level
of case can be divided by AI at first. At the same time, AI technology tracks the task points of
judges in handling cases and counts the time and frequency of each task event. Moreover, judges
can upload their personal work logs to the judges' intelligent management system synchronously
through the client of intelligent management, so as to form the judges' business files. According to
the business archives and other materials, the judge's trial features are summarized and the judge's
individual case handling time and the frequency of the case process incidents are accurately
calculated. Finally, based on the weighted calculation of the number of cases and the complexity of
the case, the true workload of the judge is calculated, and the judge is assigned a reasonable work.
Fourthly, AI is used to realize real-time supervision of the quality and effectiveness of trial work
of specified number of judges.
Through pre-determined normative indicators of court trial work, AI is used to monitor the
quality and effectiveness of court trial work in real time. On the one hand, the courts use intelligent
tools to refine and evaluate the three sub-items of the image of judges, the standard of trial
procedure and the ability of trial, so as to realize the monitoring and evaluation of the quality of trial.
On the other hand, through the live broadcast of the trial, the open trial and the relevant evaluation
of the petitioners after the trial, the effect of the trial is monitored from the perspective of the parties
and the public. Real-time monitoring of AI makes the whole process of trial behaviour and trial
management behaviour leave traces, which provides strong support for the whole trial work.
Fifth, AI is used to evaluate the quality and effectiveness of case trials and the completion of the
overall work of specified number of judges.
In the trial matters, AI is mainly used to conduct statistics, classification and analysis of the
quality, efficiency and effectiveness indicators of the judges' case handling, and to calculate the
total annual work of the judges and the courts, to achieve a comprehensive consideration of the
factors affecting the complexity of the case, the judge's personal features and capabilities, the public
opinion trend of the case effect and social evaluation, and ultimately provide a reference for the
performance evaluation of judges at the end of the year. In non-trial matters, AI is used to conduct
mathematical statistics on the number, frequency and time-consuming of individual non-judicial
matters of the specified number of judges, and finally together with the trial workload as an
indicator to measure the completion of the judge's overall work.
Sixth, AI is used to provide duty security for the specified number of judges.
Current personnel quota system reform and personnel classification management make the judge
assistant undertake a lot of matters related to the trial process, but part of the work of judicial
assistants can be entrusted to the intelligent assistant system. For example, the intelligent assistant
mentioned above provides intellectualized and integrated auxiliary services in the core judicial links,
such as reading documents, holding court sessions, writing documents and judging, and liberates the
assistant judges in order to put resources into matters more suitable for manual processing. At the
same time, AI is used to establish a reasonable safeguard system according to the professional
features of judges, to protect the right of appeal of dismissed judges. In the construction of judges'
withdrawal mechanism, AI is used to realize dynamic approval and other review procedures, as well
as follow-up relief for judges after withdrawal.

5. Method and Path of Application of AI in the Quota Management of Judges

The application of science and technology in law should be realized through specific methods
and paths. The exploration on the construction of judge quota management system may start with
the construction of multi-source heterogeneous information fusion mechanism, the establishment of
judges' business files and the establishment of case complexity evaluation model, so as to provide a
more convenient way for the dynamic quota management of judges.
First, explore the construction of multi-source heterogeneous information fusion mechanism for
judge quota management. Judge quota management is a complex work with high relevance and
dynamic. The original data of judge quota management system is collected and observed by many
different types of sensors, which mainly includes three aspects: first, various databases including
108
judgment documents, judgment cases, laws and regulations, legal journals, legal documents,
litigation data; second, the data generated by various departments in the court's business and
management; third, the data generated outside the courts include the evaluation of the parties, public
opinion, social economy, state governance and other relevant information, which are multi-source,
heterogeneous and cross-border. In order to realize the effective quota management of judges, it is
necessary to explore the construction of a multi-source heterogeneous data fusion mechanism to
transform these related multi-source heterogeneous information from different channels and in
different formats into machine-readable data, and set up a unified description method. Making use
of the features of large data, such as large capacity, multiple types and fast access speed, it can
better analyse the efficiency and correlation of legal data with dispersed sources, diverse formats
and huge quantity, so as to meet the requirements of multi-source heterogeneous information fusion.
Second, establish a business file for the specified number of judges based on judicial big data.
The use of the big data of law to establish the professional files of judges based on personnel
files and performance files can, on the one hand, achieve scientific quota management of the judges,
and on the other hand, enable judicial personnel to maintain a passion and honour for their work. In
the statistics of the basic situation of personnel, in addition to the statistics of the judge's personal
information including gender, age, education, major and length of case, it is also necessary to use
the Internet and big data to locate the identity of judges in different occasions, to understand the
individual experience and individual differences of each judge and to realize the correlation analysis.
On the one hand, it can summarize the judges' personal expertise and skills, connect the judges'
personal features with their trial features, and provide information support for the construction of
personal incentives; on the other hand, it can also provide better humanistic care for judges to
strengthen communication and trust. At the same time, the specified number of judges are
dynamically tracked and managed in the whole entry and exit mechanism by using the business
files. In the performance archives, the judges' archives are mainly divided into two aspects: trial
business and non-trial business.
In the trial business, AI is used to conduct mathematical statistics on the judge's individual case
trial type, case process matters, number of installment cases, case trial duration, and change of
judgment and return, and calculate various efficiency indicators. For non-trial business,
mathematical statistics is used to classify, count, and input basic information intelligently. Through
the integration of the above resources, and the use of AI to analyse the information of trial
management, human resources, information construction and other internal departments, the judges'
professional files are established as the basis for judges' rewards and punishments and performance
appraisal, so as to realize the multiple sharing and deep utilization of judicial data.
Third, establish a case complexity assessment model. Innovation and strengthening the quota
management of judges must be based on correct evaluation and analysis of trial work. With the
advancement of personnel quota system reform, domestic scholars have proposed a variety of
calculation models for personnel quota system research, but there are still some problems such as
lack of sample data, unreasonable weight design and so on. The evaluation of case trial quality and
effectiveness should be the focus of the whole judge management system, and the evaluation model
of case complexity is the core of the evaluation of case trial quality and effectiveness. In order to
evaluate the quality and effectiveness of a judge's case trial more directly, a case complexity
evaluation model should be constructed. There are many factors affecting the complexity of cases.
The use of mathematical models can scientifically and accurately evaluate cases, achieve the
evaluation of the quality and effectiveness of the trial of judges, and ultimately achieve the
scientific quota management of judges.

6. Technical Route of AI Applied in the Quota Management of Judges

Machine learning and other AI technologies have brought great convenience to many practical
problems, but the current application in legal scenarios is only the tip of the iceberg. For the
application scenario of judge quota management, in this paper, the following three scientific
problems are proposed and the technical routes to be solved are given.
109
6.1. Research on Feature Fusion Method Based on Multi-source Heterogeneous Information
in Law
The data in the law is complex. How to integrate, clean and align the existing huge judgment
documents, judgment cases, laws and regulations, legal journals and other data effectively is the
most important problem to be solved urgently. In the various downstream tasks in the legal scene
(such as court performance evaluation, case complexity evaluation, false litigation screening, etc.),
a variety of digital feature information is needed to model, but most of the information in the law
exists in different formats and non-standardized text or various non-quantitative forms, so it is
necessary to extract quantified digital features from these information effectively for downstream
tasks.
Specifically, information in most laws, such as party evaluation, socio-economic, regional
administrative levels, can be quantified by labelling. For most of the textual data in the law, such as
judgment documents, legal provisions, legal papers and periodicals, it is necessary to do feature
engineering for some of the textual information, vectorize the textual information and excavate the
information at all levels. From simple TF-IDF and BM25 based on word frequency to LSA and
pLSA based on shallow semantic information, and deeper network methods such as Word2Vec,
GloVe and Doc2Vec, which can represent the semantic information of words, are all methods that
can transform text in corpus into vectors. However, appropriate methods need to be chosen for
different downstream tasks.
However, for the problem of text vector embedding, even though some network-based methods
can better reflect the semantic information, they cannot solve the problem of polysemy. The
emergence of ELMo (Embedding from Language Models) solves this problem [5]. ELMo adopts
the classical two-stage model. In the pre-training stage, it uses the classical two-tier bidirectional
LSTM feature extractor. As a result, three different embedded representation vectors are obtained
for the same word, which represent the features of words, syntax and semantics respectively. The
network structure is shown in Figure 1 (Source: Devlin, J., Chang, M. W., Lee, K., & Toutanova, K.
(2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv
preprint arXiv:1810.04805.) In the downstream task, weighted sum of the three vectors can be
learned, and the different weights reflect the semantic differences of word vectors in different
contexts and tasks, which is a method of fusing features. In the legal scene, the idea of ELMo can
be considered to firstly pre-train the vector representation of the text, and then fuse the features
according to different downstream tasks.

Figure 1 ELMo Model Pre-training Network Architecture

6.2. Multi-feature-based Business Evaluation Model for Judges
For each judge, there are many dimensions in the trial task that can reflect their professional
competence. Detailed records will be made for each case of the judge's trial. Using the feature
fusion method mentioned above, important features will be extracted from these records to model
the judge's professional competence, which of course requires manual labelling of part of the data,
mainly reflected in the sample judges' ability to label their real business through comprehensive
consideration of all aspects. By learning this part of labelled data, potential rules can be grasped in
the model and then the unlabelled data can be predicted. The evaluation model can regularly
automate the assessment of judges' professional competence, and generate personal portraits of
judges, reminding judges which aspects of ability training should be emphasized.

110
6.3. Case Complexity Evaluation Model
The complexity evaluation of the case is the core issue in the evaluation of the quality of the case.
It is proposed to use AI technology to automatically judge the complexity of a case. In fact, the
main scheme adopted is to extract a part of the case features from the judgment documents to
statistically model the complexity of the case, and to request the extracted case features to
accurately reflect the complexity information of the case from at least one aspect. However, the
difficult problem is that for a case, the judgment documents are often less structured, so it is more
difficult to extract and quantify the features.

Figure 2 Technical Route of Case Complexity Evaluation Model.

After comprehensive consideration, it is proposed to adopt the following technical routes: Firstly,
the original unstructured judgment documents are structured by rule-based method, that is, the
judgment documents are sorted out in sections (for example, the name of the document, the type of
the case, the information of the parties, the fact-finding part of the court, the reasoning part of the
court, the end of the document, etc.). It is also an important part for the more complex part of the
fact-finding and reasoning of the court. It is proposed to further improve the structural accuracy of
the machine learning classification method by using the results of rule-based "pseudo-labelling".
Secondly, case features needed for modelling are extracted from semi-structured judgment
documents (e.g. the number of judges, whether they are collegial panels/summary procedures/quick
decision procedures, the number of defendants, whether they are defenders, the number of
evidences, the number of controversial focus, the length of the reasoning part of the court and the
length of the factual part of the court's determination, etc.). Thirdly, expert knowledge is used to
annotate the sample data, that is, to rate the complexity of the sample cases. Finally, various
classification methods in machine learning are used for modelling (such as logistic regression, SVM,
random forest, XGBoost and neural network, etc.). After training the above machine learning
classification model, as long as the features of the model are extracted from the unlabelled case data,
the complexity of the case can be evaluated.

Acknowledgements
This work was supported by “Development of legal big data industry from the perspective of
national governance modernization” of Sichuan Soft Science Item (18RKX0516).

References
[1] X.L. Pu. (2018). What is the status of artificial intelligence given by law?. Citizenship and the
Rule of Law. (6), 7-9.
[2] Y.L. Pan. (2018). Analysis of Paths in the Field of Artificial Intelligence Intervention in Justice.
Oriental Law. No.63 (03), 111-120.
[3] W.M. Zuo. (2018). Some Thoughts on the Application of Legal Artificial Intelligence in China.

111
Tsinghua Law Journal. 12(2).
[4] B.S. Zhang. (2001). the Jurisprudence Review of the Artificial Intelligence Legal System. Law
Review. (5), 11-21.
[5] Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L.
(2018). Deep contextualized word representations. arXiv preprint arXiv:1802.05365.

112