AVERTISSEMENT

Ce document est le fruit d'un long travail approuvé par le jury de

soutenance et mis à disposition de l'ensemble de la
communauté universitaire élargie.

Il est soumis à la propriété intellectuelle de l'auteur. Ceci

implique une obligation de citation et de référencement lors de
l’utilisation de ce document.

D'autre part, toute contrefaçon, plagiat, reproduction illicite

encourt une poursuite pénale.

Contact : [email protected]

LIENS

Code de la Propriété Intellectuelle. articles L 122. 4

Code de la Propriété Intellectuelle. articles L 335.2- L 335.10
http://www.cfcopies.com/V2/leg/leg_droi.php
http://www.culture.gouv.fr/culture/infos-pratiques/droits/protection.htm
 École doctorale IAEM Lorraine

Comptage de points de courbes

hyperelliptiques en grande
caractéristique : algorithmes et
complexité

THÈSE
présentée et soutenue publiquement le 7 septembre 2018

pour l’obtention du

Doctorat de l’Université de Lorraine

(mention informatique)

par

Simon Abelard

Composition du jury

Président : Guillaume Hanrot Professeur, ÉNS Lyon

Rapporteurs : Christophe Ritzenthaler Professeur, Université Rennes 1
Fréderik Vercauteren Associate Professor, KU Leuven
Examinateurs : Magali Bardet Maı̂tresse de Conférences, Université de Rouen
Elisa Gorla Professeure, Université de Neuchatel
Guillaume Hanrot Professeur, ÉNS Lyon
Directeurs de thèse : Pierrick Gaudry Directeur de Recherche CNRS, Nancy
Pierre-Jean Spaenlehauer Chargé de Recherche Inria, Nancy

Laboratoire Lorrain de Recherche en Informatique et ses Applications — UMR 7503

 i

À mes grands-parents.
ii
 iii

Remerciements
Théorème. Pour toute thèse en position générique, les remerciements sont la partie la plus lue
et la plus délicate à rédiger.

Preuve. C’est de notoriété publique, d’ailleurs les thèses contenant cet énoncé forment un
sous-ensemble dense de l’ensemble des thèses muni de la topologie de Zariski. 

Corollaire. Malgré mes efforts, cette partie contient son lot de formulations approximatives et
d’oublis qui sont de surcroît plus faciles à remarquer que les éventuelles erreurs tapies dans le
chapitre 4 ou dans la section 5.4.

Ces fondements théoriques solides étant posés, je sollicite l’indulgence de celles et ceux qui
me liront concernant les erreurs qui se trouvent dans mes remerciements (ou ailleurs) ainsi que
les répétitions peu élégantes du verbe remercier dans les quelques paragraphes qui suivent. J’en
profite également pour préciser que l’ordre de mes remerciements est globalement peu significatif,
même si j’ai tenté autant que possible de séparer ce qui relève du scientifique de ce qui relève
du personnel.
Mes premiers remerciements sont pour Pierrick et Pierre-Jean qui m’ont proposé un sujet de
recherche passionnant et dans lequel j’ai pu m’épanouir, notamment grâce à leur encadrement
remarquable. Merci infiniment d’avoir guidé mes premiers pas dans le monde de la recherche par
vos conseils avisés et par l’attention que vous avez portée à la préparation de mes divers exposés
ainsi qu’à la rédaction de ce manuscrit. Merci pour tout ce que vous m’avez transmis, pour votre
bienveillance et pour le plaisir que j’ai eu à travailler avec vous. Merci à Pierre-Jean pour son
enthousiasme communicatif, ses bonnes questions («affine ou homogène ?») et pour m’avoir fait
découvrir que le système polynomial c’est trop génial. Merci Pierrick d’avoir partagé ta culture
et ton expertise dans le domaine des courbes, et de m’avoir fait découvrir [28] dans lequel nous
nous sommes (re)plongés à deux reprises, avec un plaisir toujours renouvelé.
Je remercie Christophe Ritzenthaler pour avoir accepté d’être rapporteur de ma thèse, pour
l’attention qu’il y a portée ainsi que pour les remarques et les discussions enrichissantes qui
en ont découlé. I wish to thank Fréderik Vercauteren for accepting the task of reviewing my
thesis, and for his thorough reading. Many thanks also to Elisa Gorla for being a member of my
committee and attending my defense from overseas. Je remercie Magali Bardet dont la thèse
m’a beaucoup aidé à enrichir, à clarifier et à réorganiser mes connaissances en matière de bases
de Gröbner et qui a accepté de faire partie de mon jury. Merci également à Guillaume Hanrot
d’être toujours fidèle au poste dans le jury d’un énième doctorant CARAMEL / CARAMBA.
Merci enfin à Monique Teillaud d’avoir été ma référente de thèse au cours de ces trois ans.
Je remercie toute l’équipe CARAMBA pour cette ambiance agréable et stimulante : j’y
ai rencontré des gens remarquables autant par leurs compétences scientifiques et techniques
que par leurs qualités humaines. Même si j’ai dû m’habituer au troll alors que c’était tout
sauf ma spécialité, je garderai un excellent souvenir de ces années passées avec vous. Merci
d’avoir fait honneur à l’ADN Inria en encourageant ma fibre entrepreneuriale disruptive et en
supportant mes nombreux pitchs (et je ne parle pas de la brioche fourrée) et autres idées de
jeunes pousses dans des domaines allant des objets connectés pour chevaux à la blockchain du
froid. Merci également d’avoir tous contribué à ma culture scientifique mais aussi culinaire,
agricole, musicale, cinématographique et hippique. Merci en particulier à Cécile dont le chat
m’a bien aidé à rédiger l’introduction, j’espère que tu y reconnaîtras son style littéraire et que
ton cheval trouvera ça beau (vous l’avez ?).
iv

En parlant d’ambiance, je remercie tous ceux qui ont fait vivre l’esprit du bureau A215 :
Svyat, Élise, Shashank, Sandra et nos illustres prédécesseurs et notamment Hamza que je n’ai
jamais rencontré mais dont le poster et l’héritage spirituel m’ont grandement influencés. Parmi
les piliers de ce bureau, Laurent mérite des remerciements tous particuliers pour ses nombreux
conseils, ses scripts et sa grande culture. Merci également à Paul et Simon avec qui j’ai partagé
le bureau B225 pendant ma rédaction, ainsi qu’à Ludovic, Joseph, Itsaka, Ivan, Alicia et tou-
te-s les Camarades du pique-nique des doctorant-e-s pour les nombreuses conversations aussi
philosophiques qu’animées. Merci encore aux stagiaires que j’ai croisé-e-s, notamment Aude qui
commence sa thèse tandis que je termine la mienne.
Merci également à tous les collègues avec qui j’ai eu d’agréables discussions, qu’elles soient
mathématiques ou non. Il me serait difficile de tous vous citer, mais je tiens à remercier Maike
Massierer, Jan Tuitman, Cyril Hugounenq, Alexandre Gélin, David Kohel, Ben Smith, Benjamin
Wesolowski, Marius Vuille, Chloe Martindale, Daniel Lazard, Grégoire Lecerf, Reynald Lercier,
Enea Milio, mon "grand-père" Mohab et bien d’autres. Un grand merci à Éric Schost pour
son invitation à Waterloo en avril 2017 et ses efforts pour financer mon postdoc à venir. J’ai
beaucoup appris lors de ma première visite et je suis impatient de retourner à Waterloo.
Au cours de ces trois ans, j’ai aussi eu l’opportunité d’enseigner aux Mines de Nancy, ce qui
fut une expérience agréable et très enrichissante. Je remercie Antoine Henrot, pour m’avoir fait
confiance, Bernardetta Addis grâce à qui j’ai découvert et apprécié la recherche opérationnelle,
Yannick Toussaint avec qui j’ai été très content de partager bon nombre d’heures de TD, Frédéric
Sur aussi bien pour son aide précieuse concernant les subtilités administratives que pour son
expérience pédagogique ainsi que Guillaume Bonfante, Pierre-Etienne Moreau et Cédric Zanni
pour les TP de python et leur organisation bien rodée. Merci enfin à tous mes élèves pour avoir
servi de cobayes à mes expériences pédagogiques, j’espère qu’elles vous ont été profitables autant
qu’à moi et que vous en garderez un bon souvenir.
Mais l’enseignement et la recherche ne sont pas uniquement le fait des enseignant-chercheurs
eux-mêmes, aussi je voudrais remercier toutes les personnes qui m’ont grandement aidé en
gérant les aspects pratiques et administratifs associés à ma recherche. À ce titre, je remercie
Sophie, Emmanuelle, Christine, Laurence, Virginie et Françoise ainsi que tous les services de
l’école doctorale, de l’Université de Lorraine, des Mines de Nancy et du LORIA qui m’ont aidé
dans mes démarches ou qui ont contribué, parfois sans que j’en aie conscience, aux excellentes
conditions de travail dont j’ai bénéficié pendant ma thèse. Un grand merci en particulier aux
équipes du restaurant du centre (que tout le monde nous envie) et notamment à Isabelle, Tarek
et Caroline pour leur énergie et leur gentillesse.
Bien avant ma thèse, j’ai croisé la route de personnes qui m’ont encouragé à poursuivre
dans les sciences mathématiques, ou qui m’ont permis de clarifier mes projets d’études ou de
recherches. Parmi eux, je remercie Olivier Leguay et Daniel Souquet grâce à qui j’ai réalisé à
quel point les Mathématiques étaient un domaine vivant et rempli de liberté et d’opportunités.
Je remercie mes enseignants de licence et de master pour leurs conseils et plus particulièrement
Jean-Michel Morel, Bernard Landreau et Michael Harris avec qui j’ai effectué des stages qui ont
été très formateurs et qui m’ont aidé dans mon orientation. Un grand merci enfin à Célestin
Rakotoniaina qui a été mon professeur à une période charnière de mes études et qui est devenu
un ami.
Les mots me manquent pour exprimer des remerciements à la hauteur de la gratitude que j’ai
pour ma famille. Je pense notamment à mes parents qui m’ont fait grandir par leur dévouement,
leur affection inconditionnelle et l’éducation qu’ils m’ont donnée. Votre implication dans mon
parcours scolaire puis académique, votre soutien logistique à toute épreuve et la liberté que vous
m’avez laissée dans mes choix personnels ont été déterminants au point que cette thèse est aussi
 v

la vôtre.
Je dois aussi énormément à mes frères qui ont grandement contribué à l’environnement
stimulant dans lequel j’ai grandi. Merci Guillaume d’avoir été mon premier prof de maths, pour
les heures passées en jeux de stratégie qui m’ont appris la persévérance, et pour tout ce que
tu m’as transmis du haut de tes dix ans de plus. Merci Sylvain pour nos nombreux échanges
numériques de qualitay qui ont égayé mes études supérieures, pour ton hospitalité lors de mes
passages en région parisienne et pour ta relecture attentive de ces remerciements et de mon
résumé en français (on a frôlé la catastrophe !).
Je n’oublie pas non plus mes grands-parents pour tout ce qu’ils m’ont apporté. Aucun d’eux
n’aura pu voir l’aboutissement de ce travail, mais leur souvenir ne m’a jamais quitté. Cette
thèse leur est dédiée.
I am deeply grateful to my American family for the time we had the opportunity to spend
together in Atlanta and Milwaukee during my thesis. Being with you gave me the peace of mind
and the energy I needed to keep moving.
Z celého srdce děkuji paní doktorce Carole Wastiaux, která sledovala s porozuměním a
osobním pochopením mou práci v posledních pěti letech. Tato dizertační práce a její autor
jí vděčí za mnohé (merci à Lenka Froulíková pour la traduction).
Enfin, comme on dit dans My Little Pony, les amis c’est magique et c’est donc tout na-
turellement que je souhaite remercier les miens. Je pense en particulier à Ève pour notre trafic
d’animaux mignons en tous genres ; aux camarades cachanais-e-s et notamment Lilian, Pierre,
Alexandre et Édouard ; à tous mes rowers et pourvoyeurs de bons mots, d’informations inso-
lites, décalées et trollesques parmi lesquels se distinguent Rémi et mon directeur de la publication
Henri Vullierme. Merci à tous mes amis de prépa et d’avant avec qui j’ai gardé contact de façon
plus ou moins dématérialisée, notamment Déborah, Kévin et Laurène, Matthias, Thomas et
bien d’autres qui, je l’espère, ne m’en voudront pas de ne pas être nommés ici.
vi
 Contents

Introduction

Part I Background and preliminaries 1

Chapter 1 Point-counting and applications 3

1.1 Background and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.1 Abelian varieties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Curves and their Jacobians . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.3 Hyperelliptic curves and their Jacobians . . . . . . . . . . . . . . . . . 6
1.1.4 Arithmetic in hyperelliptic Jacobians . . . . . . . . . . . . . . . . . . 7
1.1.5 Endomorphisms, torsion and Tate modules . . . . . . . . . . . . . . . 9
1.1.6 Real multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2 Point-counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Applications of point-counting . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.1 Cryptographic use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.2 Extensions of the Sato -Tate conjecture . . . . . . . . . . . . . . . . . 15
1.3.3 Algorithmic applications . . . . . . . . . . . . . . . . . . . . . . . . . 16

Chapter 2 Polynomial systems 19

2.1 Solving polynomial systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 Gröbner bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.1 Gröbner bases and elimination . . . . . . . . . . . . . . . . . . . . . . 21
2.2.2 Computing Gröbner bases . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.3 Complexity results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3 Resultant-based approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.1 Resultants and elimination . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.2 Computing univariate resultants . . . . . . . . . . . . . . . . . . . . . 28

vii
viii Contents

2.3.3 Bivariate and trivariate resultants . . . . . . . . . . . . . . . . . . . . 29

2.4 Geometric resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4.1 Bézout bound and multihomogeneity . . . . . . . . . . . . . . . . . . 30
2.4.2 Geometric resolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.3 Computing geometric resolutions . . . . . . . . . . . . . . . . . . . . . 33
2.4.4 Complexity bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Chapter 3 Counting points on genus-2 curves 37

3.1 Genus-2 extensions of Schoof’s algorithm . . . . . . . . . . . . . . . . . . . . 38
3.1.1 The Gaudry-Harley-Schost algorithms . . . . . . . . . . . . . . . . . . 38
3.1.2 The case of RM curves . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 Practical improvements and past results . . . . . . . . . . . . . . . . . . . . . 43
3.2.1 Sharper modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.2.2 Further optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.3 Final collision search . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2.4 A cryptographic genus-2 curve . . . . . . . . . . . . . . . . . . . . . . 47
3.3 Prospective improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3.1 Feasability of a cryptographic 384-bit Jacobian . . . . . . . . . . . . . 48
3.3.2 Further improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.3.3 Generalization of the Elkies-Atkin improvements . . . . . . . . . . . . 49

Part II Contributions 51

Chapter 4 Cantor’s division polynomials 53

4.1 Overview on division polynomials . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2 A cubic bound in any genus . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.3 A quadratic bound in genus 3 . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Chapter 5 Asymptotic complexity bounds in arbitrary genus 65

5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2 Computing geometric resolutions . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.2.1 Main complexity result . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.2.2 Input preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.2.3 Proof of the main complexity result . . . . . . . . . . . . . . . . . . . 70
5.3 Computing generic `-torsion points . . . . . . . . . . . . . . . . . . . . . . . 71
5.4 Non-generic cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.4.1 Simple degeneracies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
 ix

5.4.2 Combining all possible degeneracies . . . . . . . . . . . . . . . . . . . 77

5.4.3 Polynomial system derived from a normalized non-genericity tuple. . 79

Chapter 6 The case of genus-3 hyperelliptic curves with RM 85

6.1 Overview of the algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.1.1 The characteristic equation of the Frobenius . . . . . . . . . . . . . . 86
6.1.2 A point-counting algorithm . . . . . . . . . . . . . . . . . . . . . . . . 87
6.1.3 Complexity overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.2 Bounds for Algorithm 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.2.1 Bounds on the coefficients of ψ . . . . . . . . . . . . . . . . . . . . . . 89
6.2.2 Small elements in ideals of Z[η] . . . . . . . . . . . . . . . . . . . . . . 90
6.3 Computing kernels of endomorphisms . . . . . . . . . . . . . . . . . . . . . . 91
6.3.1 Modelling the kernel computation by a polynomial system . . . . . . 91
6.3.2 Solving the system with resultants . . . . . . . . . . . . . . . . . . . . 92
6.3.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6.4 Practical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.4.1 Retrieving modular information . . . . . . . . . . . . . . . . . . . . . 94
6.4.2 Final collision search . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Chapter 7 Counting points on hyperelliptic curves with explicit RM 99

7.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.1.1 Families of RM curves . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.1.2 The characteristic equation . . . . . . . . . . . . . . . . . . . . . . . . 100
7.1.3 Overview of our algorithm . . . . . . . . . . . . . . . . . . . . . . . . 102
7.2 Modelling kernels of endomorphisms . . . . . . . . . . . . . . . . . . . . . . . 105
7.2.1 The generic case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.2.2 Non-generic kernel elements . . . . . . . . . . . . . . . . . . . . . . . 107
7.3 Complexity analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.3.1 Solving the polynomial systems modelling J[α] . . . . . . . . . . . . . 109
7.3.2 Dependency on g of the complexity . . . . . . . . . . . . . . . . . . . 111

Conclusion 113

Bibliography 117

Résumé en Français
x Contents
 Introduction

Curves over finite fields and their applications

Algebraic curves have been part of the mathematical landscape for over 2000 years, from the
foundations of geometry in the Antiquity to the proof of Fermat’s last theorem in the late 1990’s.
Such curves are often described as the solution set of a polynomial system and can model various
situations, hence their wide range of applications even outside mathematics. In this thesis, we
focus on algebraic plane curves, i.e. curves given by an equation of the form f (x, y) = 0 with f a
bivariate polynomial. A point of the curve corresponds to a solution of an associated equation,
but we must be clear about what we call a solution: much to their dismay, the mathematicians
of ancient Greece were confronted with the fact that even when considering equations with
coefficients in Z, the associated points may live outside Q. We must therefore specify the field
in which the coefficients of f live, which we call the base field of the curve, and consider points
in the algebraic closure of this field. Some of the points on the curve may still belong to the
base field, and they are called rational when it is the case. While the real field R seems quite
a natural field to study plane curves and in particular to plot them, curves defined over finite
fields also have many interesting applications and properties. In this thesis, we consider almost
exclusively curves defined over a finite field of odd characteristic although we sometimes take
advantage of special properties of reductions modulo primes of curves defined over Q.
Algebraic curves over finite fields can lead to efficient algorithms used in practice for factoring
integers and primality testing. Indeed, the elliptic curve method (ECM) of [93] is still competitive
compared to algorithms based on the number field sieve for finding factors of size less than
64 bits. The elliptic curve primality proving (ECPP) introduced by Goldwasser and Kilian
and improved by Atkin and Morain [65, 109] is still among the fastest algorithms to generate
primality certificates and it was used recently to prove the primality of a 34987-bit integer [76].
Although the ECCP-based algorithms are efficient in practice, the complexity of ECPP is not
proven. Using genus-2 curves, Adleman and Huang [4] designed a polynomial-time Las Vegas
algorithm for primality proving. More general curves have also been investigated to achieve a
deterministic polynomial-time algorithm for factoring polynomials over finite fields. One can
also mention the use of interpolation on algebraic curves by Chudnovsky and Chudnovsky [31]
in the late 1980’s to study the complexity of multiplying polynomials over finite fields. This
is still ongoing research and there is an important literature [118, 9] on how to improve this
method, for instance by a careful choice of the interpolating curves. To this end, the curves are
chosen to have many rational points, and [63] provides an open database of such curves over
some finite fields.
The same goes for ECM, as families of curves are chosen to increase the efficiency of the
algorithm, either because they are more likely to have a smooth cardinality [6, 10], or because
they allow for faster arithmetic [108, 16]. Elliptic curves defined over finite fields have also been
of interest to cryptographers as their rational points form a group in which computing discrete

xi
xii Introduction

logarithms is hard in general. They now represent a widespread standard which benefits from
much smaller keysizes compared to RSA. The reason is that contrary to factoring integers
or computing discrete logarithms in the multiplicative group of a finite field, there is still no
subexponential algorithm for computing discrete logarithms on an elliptic curve. Yet, a result by
Pohlig and Hellman shows in [116] that even an exponential algorithm turns out to be efficient
if the elliptic curve has a smooth number of rational points. Therefore, elliptic curves must be
carefully chosen for cryptographic applications, and in particular the number of their rational
points has to be known.
As the theory around curves developed, other objects were designed or related to curves.
Examples are the various zeta and L functions associated to curves, which are now central tools
in modern number theory. Indeed, there are various examples of number-theoretical results that
were achieved by proving analytical results for these complex functions, such as the Sato -Tate
conjecture. This conjecture gives a result on the behavior of the statistical distribution of the
number of rational points of the reduction modulo p of an elliptic curve over Q when p varies,
and was proven circa 2005 [67, 32, 139]. For more general curves, work is in progress to formulate
generalizations of the Sato -Tate conjecture such as [49]. To this end, heavy experiments are
made and point-counting represents a major part of the computations [71].
All these applications entail different contexts, from the nature of the curves involved to their
fields of definition. In this thesis, we focus on hyperelliptic curves given by an odd-degree model
y 2 = f (x), with f a monic squarefree polynomial of odd degree. The degree deg f = 2g + 1
determines the genus g of the associated hyperelliptic curve which will be an important parameter
throughout the whole manuscript. The two additional parameters p and n determine the base
field Fpn of the curve, and we set q = pn when only the size of the field matters. In the whole
manuscript, we use the usual O notation, the O e notation for the O notation in which we omit
(poly)logarithmic terms, and Og when we further omit all the terms depending only on g. Using
fast arithmetic (see for instance [24]), we assume that field operations in Fq have a cost in
O(log
e q).

Schoof’s algorithm
We have seen several reasons why knowing the number of rational points on an elliptic curve can
be crucial. One approach is to find methods to build curves with a prescribed number of points,
such as the CM-method of [6] used for cryptographic applications in [86]. Another way is to
consider “random” curves and count their points until we are satisfied with the outcome. While
there are low-brow methods for so doing, such as testing all the pairs (x, y) ∈ Fq and check if they
satisfy the curve’s equation, their complexities considerably limit their use. A groundbreaking
progress was made by Schoof in 1985, who proposed in [127] an algorithm for counting points
on elliptic curves in time polynomial in log q. Although at that time his algorithm was not
considered efficient enough for practical use, he set the path for numerous improvements and
extensions that are now known as `-adic algorithms. A few years later, Elkies and Atkin designed
improvements [128] to Schoof’s algorithm that contributed to its practicality and remarkable
efficiency. Under the name SEA (Schoof-Elkies-Atkin), the variant of Schoof’s algorithm is still
used for generating cryptographic curves and successfully addresses the problem of counting
points on elliptic curves.
The idea of Schoof’s algorithm is to compute the number of rational points modulo prime
numbers ` until the actual value can be recovered by the Chinese remainder theorem (CRT).
√
Indeed, the Weil bounds imply that it lies in an interval of size d4 qe so that the number and
maximal size of primes ` to consider is in O(log q). To obtain the modular information, Schoof
 xiii

considers the action of the Frobenius endomorphism π : (x, y) 7→ (xq , y q ) on the `-torsion, i.e.
the sets of points P such that `P is the point at infinity, which is the zero element for the addition
on the curve. For ` 6= p a prime number, the `-torsion is actually a vector space isomorphic to
(Z /` Z)2 . The action of the Frobenius endomorphism can therefore be represented by a 2 × 2
matrix, and its trace determines the number of rational points modulo `. The bottleneck of
this algorithm is the computation of π in the `-torsion, which costs O(` e 2 log q) field operations.
Taking into account the cost of such operations, the size of the largest ` and the number of
5
`, the overall complexity of Schoof’s algorithm is in O(log
e q). The SEA improvement consists
of replacing the `-torsion by a subgroup isomorphic to Z /` Z in which each operation costs
4
O(`
e log q) field operations, so that the SEA algorithm runs in time O(log e q).

Jacobians of curves
For some applications such as cryptography, the natural extension of elliptic curves are not
curves of larger genera because their rational points no longer form a group. A more suitable
tool for this purpose is to consider the Jacobian of the curve, which is a group —actually an
Abelian variety— built from formal sums of points on the curve. The same goes for the `-torsion
of an elliptic curve which has to be replaced by that of the Jacobian of the curve. In fact, we
will see that determining the `-torsion is a prominent step in order to extend Schoof’s algorithm,
and this relies extensively on arithmetic in Jacobians.
Although algorithms for group operations in non-hyperelliptic Jacobians have been designed
in [73, 83], this thesis focuses on the hyperelliptic case because it greatly simplifies the arithmetic
of the associated Jacobians, and in particular the description of the `-torsion. Elements of
genus-g hyperelliptic Jacobians can be represented by their Mumford form, which is a pair of
polynomials of respective degrees at most g and g −1. Arithmetic on elements given in Mumford
form is performed using Cantor’s algorithm [27], for a space and time complexity quasi-linear in
g log q. Through binary exponentiation, Cantor’s addition algorithm provides an efficient way
to perform scalar multiplications in the Jacobian.

Counting points on curves

In the early 1990’s, Pila [114] noticed that the theoretical machinery behind Schoof’s algorithm
still held in a much more general context. He therefore extended Schoof’s algorithm into an
algorithm for counting points on Abelian varieties and in particular on (Jacobians of) algebraic
curves. The output of Pila’s algorithm is not only the number of rational points, but the
full characteristic polynomial of the Frobenius endomorphism, or equivalently the local zeta
function of the curve. The complexity of Pila’s algorithm is still polynomial in log q but depends
on additional parameters of the input such as its genus / dimension in a much more critical way.
This algorithm was not intended to be practical but considering the particular case of genus-2
hyperelliptic curves and using suitable tools from computer algebra to describe the torsion
subgroups and the associated Frobenius action, a practical analogue of Schoof’s algorithm was
designed by Gaudry-Harley [57]. It was further improved by Gaudry and Schost to the point of
using it to generate a cryptographic genus-2 curve [60, 62]. As in genus 1, it is also possible to
build curves with a prescribed number of points, for example with the CM-method [146, 135, 44].
In the early 2000’s, other methods also based on computing the action of (p-adic approxima-
tions of) lifts of the Frobenius endomorphism were developped, first by Satoh [125] for elliptic
curves. This was later extended in a much broader context and many algorithms regrouped
under the denomination of p-adic methods were designed, considering other lifts or their actions
xiv Introduction

on different spaces. Among the vast literature on the subject, one can point another p-adic
approach for hyperelliptic curves based on Monsky-Washnitzer cohomology by Kedlaya [80] and
its counterpart in characteristic 2 by Denef and Vercauteren [41], and further extensions to more
and more general curves [30, 29, 142]. In characteristic 2, a variant of Satoh’s algorithm was
independently designed by Mestre [102], who proposed an expression of the Frobenius in terms of
an arithmetic-geometric sequence which is still the fastest option for counting points on elliptic
curves over F2n . Also in [102], Mestre suggested an extension of his method to genus 2. This
was further extending in two directions: either over field of (small) odd characteristic [94] or for
curves of larger genus [121, 95].
An interesting fact is that these methods yield practical algorithms and that their complexity
is polynomial in g and n but exponential in log p, so that both the p-adic and `-adic provide
complementary approaches when either one of p or g is small. There is still no classical point-
counting algorithm that runs in time polynomial both in g and n log p1 , but Harvey designed
in [70] an algorithm that, given a curve over Q as input, computes the zeta functions of its
reduction modulo p for all primes p of good reduction lower than a bound N . This algorithm
runs in time quasi-linear in N , meaning that the average time spent counting points on each
reduction modulo p is polynomial in log p for each p. This is particularly relevant when running
experiments for analogues of the Sato -Tate conjecture.
In this thesis we focus on the following problem, which we sometimes also call counting
points although we retrieve more information than the number of rational points on the curve
(or its Jacobian).

Computing local zeta functions of hyperelliptic curves. Given an odd

prime power q, a positive integer g and a squarefree univariate polynomial f ∈
Fq [X] of degree 2 g + 1, let C be the hyperelliptic curve with Weierstrass form
Y 2 = f (X). Compute the numerator PC ∈ Z[T ] of the local zeta function of C:
∞
!
X Ti PC (T )
Z(C/Fq , T ) = exp #C(Fqi ) · = .
i=1
i (1 − T )(1 − qT )

Where C(Fqi ) is the set of points of C whose coordinates live in Fqi .

Torsion subgroups
A key ingredient to the `-adic methods is the determination of the action of the Frobenius on
the `-torsion subgroups. In Schoof’s algorithm, the `-torsion of the input elliptic curve is the
set of points whose abscissae are the roots of the so-called `-division polynomial ψ` of degree
(`2 − 1)/2. Therefore, the action of the Frobenius endomorphism π : (x, y) 7→ (xq , y q ) on the
torsion can be computed by repeatedly squaring and reducing by the equations defining the
`-torsion: y 2 = f (x) and ψ` (x) = 0. In a more general context, Pila calls this step computing a
low-degree representation of the Frobenius.
For elliptic curves, the division polynomials give a straightforward representation of the `-
torsion. For curves of larger genera, a priori, we do not have access to a representation that
would allow us to compute a low-degree representation of the Frobenius by performing binary
exponentiation in a quotient ring. This entails an additional step in which we compute a
1
Allowing quantum primitives, such an algorithm was designed by Kedlaya in [81].
 xv

“nice representation” (e.g. a Gröbner basis) for the torsion ideal before using it to reduce the
Frobenius.
In this thesis, we follow the approach of Gaudry-Harley-Schost [57, 60, 62] and first start by
writing the equation `D = 0. To this end, we need a description of the multiplication by ` as a
rational map. For P a point of an hyperelliptic curve, there are 2g + 2 polynomials describing
the Mumford form of the divisor `(P − P∞ ) in the Jacobian. These polynomials introduced
in [28] are called Cantor’s `-division polynomials and they extend the `-division polynomial.
Writing D, an element of the Jacobian, as a sum of points, we deduce a first way of describing
the `-torsion as the solution set of the system `D = 0.
Once this first system is computed, we solve it in order to have a representation of the
`-torsion in which we can compute the action of the Frobenius endomorphism. This accounts
for most of the cost of our algorithms, both in theory and practice. We thus take particular
care of the way we model the `-torsion by polynomial systems and the techniques we use to
solve them, as they have a significant impact on the final complexities and running time of our
point-counting algorithms.

Solving polynomial systems

Given multivariate polynomials f1 , . . . , fm in K[x1 , . . . , xn ], we want to find equations defining

the ideal I = hf1 , . . . , fm i such that it becomes possible to perform arithmetic operations in
K[x1 , . . . , xn ]/I. In this thesis, the task is achieved for I = I` , the `-torsion ideal of a Jacobian
by either computing a triangular form of the polynomial system f1 = 0, . . . , fm = 0, or a
geometric resolution, i.e. a parametrization of the coordinates of the solutions by the roots of a
univariate polynomial. In both cases, we refer to this as solving the input polynomial system.
The literature provides numerous ways of doing so, and we will use three of them depending on
the complexity or performance they offer. This depends on many parameters such as the number
of variables and the degrees of the polynomials fi , but also on less conspicuous properties of
the system such as its dimension, its degree (i.e. its number of solutions in K̄ if it is finite) and
some potential structural particularities. In this thesis, we solve systems that model subsets of
`-torsion subgroups, so we already know that they are zero-dimensional and we can bound their
degrees by `2g .
For instance, the torsion of genus-2 curves involve bivariate polynomials for which a triangu-
lar form can be computed using bivariate resultants, which is currently the best option both in
terms of asymptotic complexity and practical efficiency. In Chapter 6, we model the `-torsion
of a genus-3 hyperelliptic Jacobian by a trivariate polynomial system, which is put in triangular
form by computing resultants. Although this gives satisfactory asymptotic complexity bounds,
computing a Gröbner basis with the F4 algorithm [45] is much more efficient in practice so we
used it instead of the resultants for practical experiments. However, although the complexity
of the F4 and F5 algorithms [45, 46] are subject of thorough investigations [11, 12], none of the
existing complexity bounds were sharp enough for us to use them both in theory and practice.
In Chapter 5, we model the `-torsion of hyperelliptic curves of arbitrary genus in a different
way, involving O(g 2 ) variables instead of g. However, there are only g variables whose degree
depends on `, while the others have degrees in Og (1). In this case, the choice of the geometric
resolution algorithm of [26, 64] was dictated by the necessity of invoking complexity results that
take advantage of this particular multihomogeneous structure.
xvi Introduction

Contributions
This thesis focuses on `-adic methods derived from Schoof-Pila’s algorithm. A central question
of the whole manuscript is the complexity of such methods and in particular the dependency
on g of the exponent of log q. The first contribution of this manuscript, to appear as [1], is a
point-counting algorithm for hyperelliptic curves, whose complexity is such that this exponent
asymptotically grows linearly in g when the characteristic p is large enough. This improves
on previous results by Adleman and Huang [3] who proved that this exponent was in general
polynomial in g and even quadratic in the case of hyperelliptic curves. The state of the art
concerning this exponent is detailed in Table 1. To achieve this complexity result, our algorithm
itself is no different from that of Pila but our complexity analysis benefits from a novel modelling
of the `-torsion by a structured polynomial system, as explained above. This structure is the key
of the improvement, and performing  our analysis without
 exploiting it yields a result similar to
2
that of Adleman and Huang in O (log q)O(g log g) . This involves some technicalities, however,
as we must first ensure that our system satisfies some genericity hypothesis to invoke complexity
bounds for the computation of a geometric resolution; also, our modelling involves in fact many
polynomial systems to handle “special” torsion elements.

Table 1: Asymptotic complexity bounds for computing the local zeta function of a g-dimensional
Abelian variety defined over Fq

Authors (year) Complexity Context

 O(g)

Pila [114] (1990) O (log q)g Abelian varieties
 
Huang-Ierardi [75] (1998) O (log q) g O(1) Plane curves
 
Adleman-Huang [3] (2001) O (log q) g O(1) Abelian varieties
 
Adleman-Huang [3] (2001) O (log q) O(g 2 log g) Hyperelliptic curves
 
Chapter 5 (2017) Og (log q)O(g) Hyperelliptic curves
8


Chapter 7 (2018) O
e η (log q) Hyp. curves with explicit RM

Another aspect we study is the practicality of Schoof-Pila’s algorithm in small genus, which
goes along with the value of the exponent of log q for a fixed genus. Although Pila’s algorithm
seems unfit for straightforward implementation, what he calls a small representation of the
Frobenius, i.e. the Frobenius modulo the `-torsion ideal can be computed in practice using
standard tools from computer algebra. This was studied and implemented in genus 2 by Gaudry,
Harley and Schost in [57, 60, 62]. Due to the size of the objects to manipulate, the complexity is
much larger than in genus 1 but the algorithm is practical enough so as to provide a cryptographic
curve defined over a 128-bit prime field. In this thesis, we informally analyze the feasibility of
designing such a secure genus-2 curve over a field of 192-bit characteristic, which seems quite
unlikely at the moment. Curves equipped with an explicit and efficient real multiplication (RM)
benefit from additional structure that is used in [59] to decrease the exponent of log q from 8 to
5, reaching a complexity similar to that of Schoof’s algorithm.
One step further, the other main contribution within this manuscript deals with hyperelliptic
curves of genus 3 [2]. Practical experiments in that case seem almost hopeless for primes ` > 3.
 xvii

However, for genus-3 hyperelliptic curves with explicit RM, the work of [59] extends modulo
6
several additional subtleties with a complexity in O(loge q), even lower than that of the general
genus-2 case. As expected from such a result, the algorithm is quite practical, although efficiency
requires some modifications compared to the version used to establish the complexity bound.
In particular, we count points on a genus-3 hyperelliptic curve with RM defined over the prime
field F264 −59 , which has a 192-bit Jacobian. Our algorithm can readily be turned into a point-
counting algorithm for general genus-3 hyperelliptic curves (i.e. without explicit RM) with a
14
much larger complexity in O(loge q), thus giving a partial answer for the complexity of the
Schoof-Pila algorithm in genus 3. As in the genus-2 case, the bottleneck of our algorithm is
the resolution of the polynomial system describing the `-torsion. This system is trivariate but
successive elimination using resultants is still sufficient to achieve our reference complexity which
is the square of the degree of the ideal. In practice however, we computed a Gröbner basis using
the F4 [45] and FGLM [47] algorithms because they were far more efficient, although their
theoretical complexity is much harder to control in our case.
Since the literature presents numerous examples of RM-curves of any genus [87, 23, 43,
101, 138], it is quite natural to wonder what changes this additional structure brings to the
asymptotic complexity when g is no longer fixed to 2 or 3. We therefore extended some results
and methods of the genus-3 case to design a point-counting algorithm for hyperelliptic curves
with explicit RM of arbitrary high genus. The main primitive we use is the computation of a
geometric resolution for the kernel of an endomorphism of degree `2 . This is done by adapting
the machinery of Chapter 5 which was applied to the kernel of the multiplication by `, itself
being an endomorphism of degree `2g . The difference of degrees impacts our modelling by
reducing the degrees of the equations from Og (`3 ) to Og (`3/g ). Therefore, after checking that the
hypotheses still hold and applying the geometric resolution algorithm, we achieve a complexity
in Og ((log q)c ), with c an absolute constant and Og hiding a term that depends both on g and the
ring by which the curve has RM. However, we emphasize that our algorithm is not polynomial
both in g and log q because the factor hidden by the Og -notation remains exponential in g. We
nonetheless analyze the cause of that exponential dependency in the hope that further results
might provide tighter complexity estimates for the exponential steps, or find a way to replace
or remove them.

Table 2: Asymptotic complexities for computing the local zeta function of hyperelliptic curves
of genus ≤ 3

Genus Complexity Authors (year)

5
g=1 O(log
e q) Schoof [127] (1985)
4
g=1 O(log
e q) Schoof-Elkies-Atkin [128] (∼ 1990)
8
g=2 O(log
e q) Gaudry-Harley-Schost [62] (∼ 2000)
14
g=3 O(log
e q) Chapter 6 (2018)
5
g = 2 with RM O(log
e q) Gaudry-Kohel-Smith [59] (2011)
6
g = 3 with RM O(log
e q) Chapter 6 (2018)
xviii Introduction

Organization of the thesis

Chapter 1 gives definitions and an overview on (hyperelliptic) curves, arithmetic in their Jaco-
bians and point-counting. We survey in deeper details some applications of point-counting and
recall fundamental results that lie at the heart of the Schoof-Pila algorithms. Since modelling
the `-torsion by polynomial systems and controlling their degrees and structures are cornerstones
of our contributions, Chapter 2 presents three techniques for solving polynomial systems, along
with their complexities, that will be used in all the following chapters apart from Chapter 4.
Chapter 3 reviews previous work on point-counting over genus-2 curves, and finishes with an
updated analysis on prospective and change that occurred in the last years. Although most of
its content was produced before this thesis, we emphasize on the parts that are later reused or
adapted.

Chapter I Chapter II Chapter IV

Background on curves Polynomial systems Cantor’s polynomials

Chapter III
Genus-2 curves

Chapters V and VI
Genus-g and genus-3 with RM

Chapter VII
Genus-g curves with RM

Figure 1: Chapters’ dependencies

Chapter 4 provides bounds on Cantor’s analogue to `-division polynomials, which we use to

bound the degrees of the systems modelling the `-torsion. These bounds were originally proven
in [1, Sec. 6] and [2, Sec. 6] but we regrouped them to form a chapter that does not rely on any
other, and that can be skipped on first reading. Indeed, the results are restated when needed
so that a reader willing to skip the proofs can avoid to read this technical chapter. Chapter 5 is
based on [1] and presents a probabilistic algorithm for counting points on hyperelliptic
 curves

over fields of sufficiently large characteristic with time and space complexity in Og (log q)O(g) .
Chapter 6 follows [2] and deals with point-counting on genus-3 hyperelliptic curves, mostly for
 xix

curves with an explicit RM. Lastly, Chapter 7 combines the approaches of both Chapters 6
and 5 to improve the bounds of Chapter 5 in the case of hyperelliptic curves with explicit real
multiplication (RM). To this end, we extend the algorithm and results of Chapter 6 in any genus
and then prove that the systems involved in the extended algorithm satisfy genericity hypotheses
similar to those of 5, so that the complexity bounds for computing a geometric resolution of
these systems still apply. The complexity gain over the general case is then a pure consequence
of the smaller degrees of the systems involved. Figure 1 sums up the dependencies between all
the chapters.
xx Introduction
 Part I

Background and preliminaries

1
 Chapter 1

Point-counting and applications

In this chapter, we introduce objects and concepts of algebraic geometry that are ubiquitous
in this thesis such as curves, Jacobians and point-counting. We also recall fundamental results
used by point-counting algorithms such as the Weil conjectures. Section 1.2 reviews the main
families of point-counting algorithms and their principles, and Section 1.3 presents applications
of point-counting.
In the whole manuscript, p stands for a prime number and q = pn is a power of that prime.
We denote by Fp the finite field of cardinality p and by Fq its extension of degree n, up to
isomorphism. In this first chapter, we consider objects (curves and varieties) defined over a
perfect field K which will often, but not always, be a finite field in the other chapters. We
denote by K̄ the algebraic closure of K.

1.1 Background and definitions

1.1.1 Abelian varieties
Definition 1.1. We denote by Pn (K̄) the quotient set {(X0 : X1 : · · · : Xn ) | Xi ∈ K̄, ∃ j, Xj 6=
0}/ ∼, for the equivalence relation

(X0 : X1 : · · · : Xn ) ∼ (Y0 : Y1 : · · · : Yn ) ⇔ ∃λ ∈ K̄, ∀i Xi = λYi .

Definition 1.2. We define An (K̄) = {(x1 , . . . , xn ) | xi ∈ K̄} the set of affine points.

For an extension L ⊂ K̄ of K, its absolute Galois group Gal(K̄/L) acts coordinate-wise on

Pn (K̄) and we define the set of L-rational points Pn (L) as the subset of Pn (K̄) fixed by this
action. The same can be done to define An (L) the set of L-rational points of An (K̄). In other
words, we have

Pn (L) = {(X0 : X1 : · · · : Xn ) ∈ Pn (K̄) | ∃λ ∈ K̄ ∀i, λXi ∈ L},

and
An (L) = {(x1 , . . . , xn ) | xi ∈ L}.
Both the affine and projective spaces can be endowed with the Zariski topology, for which
we refer to [68, Chap. 1, Sec. 1 and 2]. A subset of Pn (K̄) (resp. An (K̄)) is closed for the
Zariski topology if and only if it is the set of simultaneous zeroes of homogeneous polynomials
in K̄[X0 , . . . , Xn ] (resp. of polynomials in K̄[x1 , . . . , xn ]).

3
4 Chapter 1. Point-counting and applications

For S a set of polynomials in K̄[X0 , . . . , Xn ] (resp. K̄[x1 , . . . , xn ]), we denote Z(S) the
associated closed set in Pn (K̄) (resp. An (K̄)).
Let V be a Zariski closed subset of either Pn (K̄) or An (K̄), and IK be the associated ideal
of (homogeneous) polynomials of either K[X0 , . . . , Xn ] or K[x1 , . . . , xn ] vanishing on V . We say
that V is defined over K if and only if Z(IK ) = V .
If IK is a prime ideal, we say that V is irreducible over K. Note that irreducibility depends
on the√field K as, for instance, the ideal I = hx21 − 2x22 i is a prime ideal in Q[x1 , x2 ] but it splits
in Q( 2)[x1 , x2 ]. When IK̄ is a prime ideal, we say that V is absolutely irreducible.

Definition 1.3. A projective (resp. affine) variety over K is an irreducible projective closed set
over K.

Definition 1.4. The dimension dim(V ) of a variety V is the largest integer k such that there
exist a chain S0 ) S1 ) · · · ) Sk of subsets of V that are closed and absolutely irreducible. A
variety of dimension 1 is called a curve.

Definition 1.5. Let V ⊂ An (K̄) be an affine variety over K. It corresponds to a prime ideal
I(V ) = {f ∈ K[x1 , . . . , xn ] | ∀P ∈ V, f (P ) = 0}. Denote K[V ] = K[x1 , . . . xn ]/I, since it is an
integral domain we can define its quotient field K(V ). The ring K[V ] and the field K(V ) are
respectively called the coordinate ring and function field of V .
For V a projective variety, defining I(V ) as the set of homogeneous polynomials vanishing
on V , we similarly define the notion of coordinate ring K[V ] and we define K(V ) as the set of
quotients of homogeneous polynomials of identical degrees.

Definition 1.6. [34, Def. 4.33 & 4.34] A morphism ϕ from An (K̄) to A1 (K̄) is given by a
polynomial f ∈ K[x1 , · · · , xn ] and defined by ϕ : P = (a1 , . . . , an ) 7→ f (a1 , . . . , an ) = f (P ).
Likewise, a morphism between An (K̄) and Am (K̄) is given by a m-tuple of polynomials in
K[x1 , · · · , xn ].

Definition 1.7. [34, Def. 4.35] A K-rational morphism between two affine varieties V ⊂
An (K̄) and W ⊂ Am (K̄) is defined as a morphism ϕ : An (K̄) → Am (K̄) between their associated
affine spaces such that ϕ(V ) ⊂ W .

Definition 1.8 (Rational map). [34, Def. 4.40] Let U be a nonempty open set of an affine
variety V , a rational map from V to A1 (K̄) with definition set U is a map rU : U → A1 (K̄)
given by rU (P ) = ψ(P )ϕ(P )−1 for some ψ, ϕ ∈ K̄[V ] such that ϕ does not vanish on U .

We say that two rational maps are equivalent if they coincide on the intersection of their
respective definition sets. This defines an equivalence relation whose classes are called rational
functions.

Proposition 1.9. [34, Prop. 4.42] Let V be an affine variety, the set of rational functions on
V is a field which is isomorphic to its function field K(V ).

Definition 1.10 (Regularity at a point). [34, Def. 4.48] A rational function f ∈ K(V ) is regular
at a point P ∈ V if it has a rational map with set of definition containing P as a representative.

As in Definition 1.6, considering tuples of rational maps and functions, these notions extend
to rational maps and functions between varieties.
Replacing polynomials by homogeneous polynomials, and affine spaces by projective spaces,
rational maps, rational functions and regularity are similarly defined for projective varieties.
1.1. Background and definitions 5

Definition 1.11. [34, Def. 4.53] An algebraic group G over K is an absolutely irreducible variety
defined over K, along with

• a K-rational morphism ⊕ : G × G → G for the group law,

• a K-rational morphism ι : G → G for the inverse,

• a K-rational point 0 ∈ G(K) for the neutral,

such that ⊕ is associative, 0 is the neutral element for ⊕ and for any e ∈ G, ⊕(e, ι(e)) = 0.

For L an extension of K, denote G(L) the set of L-rational points, it is a group in which the
group law is computed by evaluating the previous morphisms that are defined on K and do not
depend on L.
Surprisingly, when G is a projective variety one can prove that the group law induced by ⊕
has to be commutative, leading to the following definition:

Definition 1.12. An Abelian variety over a field K is a projective algebraic group over K.

1.1.2 Curves and their Jacobians

In general, Abelian varieties are not easy objects to manipulate, as representing their elements
may require a number of coordinates that is exponential in the dimension. See for instance [96]
for group laws in Abelian varieties using theta functions. However, many examples of Abelian
varieties come from simpler cases, for which this difficulty can be avoided. Let us now focus on
a particular class of Abelian varieties: Jacobians of curves.

Definition 1.13. Let P be a point on a curve C. The set of rational functions that are regular
at P is a subring of K(C) denoted OP .

Definition 1.14. A point P on a curve C is called nonsingular if OP is integrally closed. We

say that P is singular otherwise and that C is a nonsingular or smooth curve if every point of
C(K̄) is nonsingular.

From now on, the word curve will refer to a smooth projective curve unless mentioned
otherwise.

Definition 1.15. Let C be a smooth projective and absolutely irreducible curve over K. The
free Abelian group with basis C(K̄) is called the divisor group of C, written DivC . An element D
of DivC has the form X
D= nP P,
P ∈C(K̄)

where the nP are integers such that only a finite number of them are non-zero. We define
Supp(D) the support of D as the set of points P such that nP 6= 0 and the degree of D as
P
deg D = P ∈C(K̄) nP .

Definition 1.16. The set of degree-zero divisors forms a subgroup of DivC that we denote Div0C .
P
Definition 1.17. A divisor D = P ∈C(K̄) nP P is said to be effective if for all P we have
nP ≥ 0, and for D and E two divisors, we write D ≥ E if D − E is effective.
6 Chapter 1. Point-counting and applications

Definition 1.18. Let L be an intermediate field between K and K̄, the action of Gal(K̄/L) on
C(K̄) induces an action on DivC (resp. Div0C ). We define DivC (L) (resp. Div0C (L)) the subgroup
of L-rational divisors (resp. degree-zero divisors) as the subgroup of DivC (resp. Div0C ) fixed
under that action.

Definition 1.19. Let L be an intermediate field between K and K̄, let ϕ be a non-zero rational
function in L(C) and set vP (ϕ) equal to either the multiplicity of P as a zero of ϕ, minus its
multiplicity as a pole of ϕ or zero if P is neither a pole nor a zero of ϕ. We define the associated
P
divisor as (ϕ) = P ∈C(K̄) vP (ϕ)P . A divisor of this form is said to be principal, and we denote
PrC (L) the group of principal divisors.

Remark that a principal divisor has to be in Div0C ([34, Prop. 4.104]), which allows the
following definition:

Definition 1.20. Let L be an intermediate field between K and K̄, we define the degree-zero
Picard group of C as the quotient Pic0C (L) = Div0C (L)/ PrC (L).

Definition 1.21. Let D be a divisor. We define the associated Riemann-Roch space as

L(D) = {ϕ ∈ K(C) | (ϕ) ≥ −D}.

This is a vector space over K whose dimension is denoted `(D).

Theorem 1.22 (Riemann’s inequality). Let C be as in Definition 1.15. Then there exists an
integer g ≥ 0 such that for any D ∈ DivC ,

`(D) ≥ deg D − g + 1.

The smallest such g is called the genus of the curve C.

Theorem 1.23. [106, Th. 1.1 & Prop. 2.1] Let C be a smooth projective and absolutely irre-
ducible curve of genus g > 0 over K and L/K an extension. Then, there exists an Abelian variety
J of dimension g over K such that J(K) = Pic0C (K̄)Gal(K̄/K) and such that J(L) = Pic0C (L) as
soon as C(L) 6= ∅. This Abelian variety J is called the Jacobian (variety) of the curve C, and it
is denoted either Jac C or JC .

1.1.3 Hyperelliptic curves and their Jacobians

Performing explicit group operations in Jacobians of curves has drawn a lot of attention and
many algorithms were proposed to achieve this goal with a polynomial-time complexity in g, such
as [83, 73]. However, since the contributions presented in this thesis only apply to hyperelliptic
curves, we do not give further detail on those algorithms and restrict to one of the simplest
example of Abelian varieties: hyperelliptic Jacobians. In particular, following [27], we present a
way to store elements of such Jacobians using O(g) elements in the base field, and an algorithm
to add points in the Jacobian in time quasi-linear in g.

Definition 1.24. An elliptic curve over K is a nonsingular absolutely irreducible projective

curve of genus 1 over K with at least one K-rational point.

Definition 1.25. A nonsingular projective curve C of genus g > 1 over K is called a hyperelliptic
curve if there exists a function x ∈ K̄(C) such that the function field K(C) is a separable quadratic
extension of the rational function field K(x).
1.1. Background and definitions 7

By [34, Theorem 4.122], if we characterize hyperelliptic curves by their affine plane parts,
we can rewrite the previous definition in a more concrete way:
Definition 1.26. Let K be a field of characteristic 6= 2, any plane affine curve given by an
equation of the form
C : y 2 = f (x),
with f in K[x] such that f is monic of degree 2g + 1 and squarefree is birationally equivalent to a
hyperelliptic curve of genus g over K. Such hyperelliptic curves are called imaginary hyperelliptic
curves.
In the remainder of this thesis, we will sometimes refer to “the hyperelliptic curve C of
equation y 2 = f (x)”. To be accurate, this refers to the nonsingular projective curve birationally
equivalent to C which is indeed a hyperelliptic curve in the sense of Definition 1.25.
Note that when setting g = 1 in the equations of imaginary hyperelliptic curves, we fall back
to the case of elliptic curves, which are famous for their use as cryptographic groups (i.e. groups
in which the discrete logarithm problem is hard). Curves of genus 2 are no longer groups but
their Jacobians also offer good candidates for cryptosystems, in a sense that we detail later on.
The first requirement for constructing a cryptographic group is to provide an efficient way to
represent and manipulate its elements: this is achieved thanks to the Mumford form for divisors
and Cantor’s algorithm to add and reduce them. Before giving details on this, we first review
the specificities of hyperelliptic Jacobians.
Like elliptic curves, imaginary hyperelliptic curves have a unique K-rational point P∞ at
infinity and an involution sending an affine point (x, y) to its opposite (x, −y). In what follows,
we see that these additional properties give a simpler description of divisors on C.
Definition 1.27. Let C be a hyperelliptic curve and D = P ∈C(K̄) nP P be a divisor in Div0C . We
P

say that D is semi-reduced if for any P 6= P∞ we have nP ≥ 0 and nP n−P = 0. Furthermore,

we say that a semi-reduced divisor D is reduced if P 6=P∞ nP ≤ g.
P

Theorem 1.28. Any element of Pic0C is uniquely represented by a reduced divisor.

The following theorem gives an efficient way of manipulating elements of JC which is used
in computer algebra systems.
Theorem 1.29. Let C be a hyperelliptic curve of genus g given by an equation of the form
y 2 = f (x) with f a monic squarefree polynomial of degree 2g + 1. Each element of Pic0C (K) can
be represented by a unique pair of polynomials u, v ∈ K[x] where u is monic, deg v < deg u ≤ g
and u|v 2 − f . The pair hu, vi is called the Mumford form of the divisor class.
The link between the previous two representations is the following. If an element of Pic0C is
represented by a reduced divisor ri=1 (Pi − P∞ ) where each Pi has coordinates (xi , yi ), then its
P

Mumford is hu, vi with u of degree r whose roots are the xi ’s counted with multiplicities and v
satisfying v(xi ) = yi . The integer r ≤ g is called the weight of the divisor.
From the group isomorphism of Theorem 1.23 between Pic0C (L) and JC (L) for any K ⊂ L ⊂
K̄, the Mumford form also gives a way of representing the points of the Jacobian of C.

1.1.4 Arithmetic in hyperelliptic Jacobians

Algorithm 1, originally described by Cantor [27] in odd characteristic and later extended by
Koblitz [85] to arbitrary fields, performs additions of reduced divisors given in Mumford form.
8 Chapter 1. Point-counting and applications

input : Two reduced divisors D1 = hu1 , v1 i and D2 = hu2 , v2 i on the curve

C : y 2 = f (x), given in Mumford form.
output: The unique reduced divisor D = D1 ⊕ D2 .
Composition step:
Compute d1 = gcd(u1 , u2 ) and e1 , e2 such that d1 = e1 u1 + e2 u2
Compute d = gcd(d1 , v1 + v2 ) and c1 , c2 such that d = c1 d + c2 (v1 + v2 )
s1 ← c1 e1 , s2 ← c1 e2 , s3 ← c2
u ← ud1 u2 2 , v ← s1 u1 v2 +s2 u1 vd1 +s3 (v1 v2 +f ) mod u
Reduction step:
while deg u > g do
2
U ← f −v u , V ← −v mod U
u ← U, v ← V
end
Make u monic
return u,v
Algorithm 1: Cantor’s algorithm

e 2 ), but this complexity can be reduced to

The algorithm we describe has a complexity in O(g
O(g)
e by replacing the reduction step by a more efficient one inspired by the fast gcd algorithm.
Since performing group operations in JC is essential, faster algorithms have been designed in [90,
88, 55] to reduce the number of field operations involved in less general frameworks. In what
follows, we make use of some of them in genus 2 and 3.
Another fundamental operation is scalar multiplication of a divisor. Once the addition is
known, this can be done by a double-and-add approach but we emphasize here on the form of
the result rather than the method to achieve it. In the case of elliptic curves, given an affine
point P ∈ C of coordinates (x, y) and an integer ` > 1, we have
!
ψ`−1 ψ`+1 (x) ψ2` (x, y)
`P = x− , ,
ψ`2 (x) 2ψ`4 (x)

where the ψi ’s are called division polynomials and they are defined inductively by ψ0 = 0, ψ1 = 1
and
3 2
ψ2m+1 = ψm+2 ψm − ψm−1 − ψm−2 ψm+1 for m ≥ 2,
ψm 
2 2

ψ2m = ψm+2 ψm−1 − ψm−2 ψm+1 for m ≥ 3.
2y
These polynomials have been generalized in [28] as follows: given ` > g and the weight-one
divisor D = P − ∞ with P ∈ C of coordinates (x, y) the generic point, there exist 2g + 2
polynomials (di )0≤i≤g and (ei )0≤i≤g such that the Mumford form of `D is
* g−1 g−1 +
X di (x) i X ei (x) i
Xg + X ,y X .
i=0
dg (x) e (x)
i=0 g

As in the elliptic case, there exist recurrence formulas for those division polynomials, which
we use later to bound their degrees. To compute them, however, it is much simpler to directly
multiply the generic affine point (x, y) by ` in the function field of the curve. In Chapter 4, we
present two bounds for the degrees of Cantor’s division polynomials, one for hyperelliptic curves
of arbitrary genera and another sharper bound specific to genus-3 hyperelliptic curves.
1.1. Background and definitions 9

1.1.5 Endomorphisms, torsion and Tate modules

To be more precise about what is point-counting and the main algorithms to do so, we give fur-
ther theoretical background. We switch back to a broader context as we later present algorithms
capable of counting points on Abelian varieties.

Definition 1.30. Let A and B be two Abelian varieties over K, and let ϕ ∈ HomK (A, B) be a
morphism of Abelian varieties, i.e. a morphism of varieties that is also a group homomorphism.
We say that ϕ is an isogeny if the induced morphism A(K̄) → B(K̄) is surjective and has a
finite kernel. If there exists such an isogeny, we say that A and B are isogenous.

Definition 1.31. The degree of such an isogeny is defined as its degree as a rational map.

Definition 1.32. Given an isogeny ϕ of degree n between A and B, there exists a unique isogeny
ϕ∨ of degree n between B and A such that ϕϕ∨ = [n]. We call it the contragredient isogeny of
ϕ.

Definition 1.33. The set HomK (A, A) of endomorphisms of A, denoted EndK (A), is a ring
with composition as a multiplicative structure, called the endomorphism ring of A.

Example 1.34. Let A be an Abelian variety over Fq . Let π be the Frobenius map x 7→ xq of
Fq , it extends to a map of projective spaces which stabilizes A, since A is defined over Fq . The
group law and zero-element of A are also defined over Fq so π is also an endomorphism for the
group structure of A. Thus, π ∈ EndFq (A) can be seen as an endomorphism called the Frobenius
endomorphism.

Another common endomorphism is the aforementioned scalar multiplication that we denote

[`]. We say that an element of ker[`] is an `-torsion point and denote A[`] the `-torsion, i.e. the
elements of A(K̄) that vanish after multiplication by `. This set is at the heart of Schoof-like
algorithms and so is the following statement about its structure.

Proposition 1.35. [34, Th. 4.73] Let A be an Abelian variety of dimension g defined over K
of positive characteristic, and let n be an integer coprime to the characteristic of K. Then A[n]
is a Z /n Z-module isomorphic to (Z /n Z)2g .

Note that it is important to highlight the fact that we consider the torsion elements in the
algebraic closure, for they have no reason to be rational, and in general they live in (large)
extensions of the base field.
In what follows, let ` be a prime number different from the characteristic. For any positive
k, [`]A[`k+1 ] = A[`k ]. Thus, the groups A[`k ] form a projective system, which brings us to the
following definition.

Definition 1.36. Let ` be a prime different from char(K), the `-adic Tate module of A is defined
as T` (A) = lim A[`k ].
←−
We have seen that for n coprime to char(K), A[n] has a structure of free Z /n Z-module of
dimension 2g, from which we deduce that T` (A) is a free Z` -module, also of dimension 2g. Thus,
Aut(T` (A)) and Aut(A[n]) can be respectively identified with GL2g (Z` ) and GL2g (Z /n Z).
By acting on each A[`k ], the Frobenius endomorphism acts on the Z` -module T` (A), and we
can extend its action to the 2g-dimensional Q` -vector space T` (A) ⊗Z` Q` . This action can be
represented by a square matrix of size 2g whose characteristic polynomial we denote χ` .
10 Chapter 1. Point-counting and applications

Theorem 1.37. [34, Lem. 5.71] The polynomials χ` have integer coefficients which are inde-
pendent from `. Their common value χ is called the characteristic polynomial of the Frobenius
endomorphism.

Note that this section relies on some powerful theoretic results that we do not want to linger
on. In what follows we will mostly consider actions of the Frobenius on subspaces such as the
`-torsion, on which there are more elementary definitions. We invite the interested reader to
look for more detailed information on this subject in [111, Sec. 19].

1.1.6 Real multiplication

Definition 1.38. We say that an Abelian variety is simple when it has no proper non-zero
Abelian subvariety.

Proposition 1.39. [137] Denote End0K (A) = EndK ⊗Z Q. If A is simple, then End0K (A) is a
skew field.

Definition 1.40. Let F be a totally real number field, we say that A has real multiplication
(RM) by F if there exists an embedding F ,→ End0K (A).
Likewise, we say that A has RM by a subring R of a totally real number field if there exists
an embedding R ,→ EndK (A).

Previous examples of endomorphisms highlighted the fact that if A is nonzero, then Z is

always a subring of the ring EndK (A). When K is a finite field, this is also true for Z[π] with
π the Frobenius endomorphism.
This definition may seem tautological as every Abelian variety has RM by Z. In Section 3.1.2
and Chapter 6, we ask for hyperelliptic Jacobian with RM by an order Z[η] satisfying further
constraints.

1.2 Point-counting
1.2.1 Definitions
Definition 1.41 (Local zeta function). Let C be a nonsingular projective algebraic curve over
a finite field Fq , the (local) zeta function of C is defined as the formal power series in Q[[t]]:
 
X tk
Z(t) = exp  #C(Fqk ) .
k≥1
k

In what follows, most point-counting algorithms actually compute the whole zeta function of
the input curve instead of simply computing #C(Fq ) or #JC (Fq ). The reason is that these algo-
rithms strongly rely on the fact that zeta functions satisfy remarkable properties, as conjectured
by Weil in 1949 and later proved by Dwork, Grothendieck and Deligne.
The Weil conjectures can be summed up by the following three properties:

• Z(t) ∈ Q [[t]] is a rational function

• Z(t) verifies a functional equation

• the numerator of Z(t) is a polynomial in Z[t] whose roots are algebraic integers of norm
√
1/ q.
1.2. Point-counting 11

For counting points, we use the following consequences of the Weil conjectures.

Proposition 1.42. The zeta function Z(t) of a nonsingular projective algebraic curve C is a
rational fraction of the form
L(t)
Z(t) = ,
(1 − t)(1 − qt)
where L = a0 + · · · + a2g t2g is a degree 2g polynomial whose coefficients ai are integers such that
a0 = 1, a2g = q g and !
g−i 2g i/2
∀i ≤ g, a2g−i = q ai and |ai | ≤ q .
i

We have reduced the problem to computing the polynomial L, but we can get even more
information on this polynomial by relating it further to the curve. In fine, we can translate all
these properties into efficient point-counting algorithms.

Lemma 1.43. The polynomial L is the reciprocal polynomial of the characteristic polynomial χ
of the Frobenius endomorphism, as defined in Definition 1.37.

This lemma is the cornerstone to all the algorithms that we discuss next, each one follows
the same principle indeed: deducing χ from the characteristic polynomial of the action of the
Frobenius on some spaces, of course provided that we can recover the actual χ from the partial
information obtained.

1.2.2 Algorithms
In this section, we review the main families of algorithms for counting points on hyperelliptic
curves, as well as their complexities. Note that the input curve is given by a degree 2g + 1
polynomial in Fq [X], with q = pn . This input has a bit-size in O(ng log p) which is why ng log p =
g log q is the reference when we give complexity estimates. Therefore, an algorithm in O(p) will
be called exponential. See for instance [56] for a survey on the subject, along with record
computations.

Exhaustive search
Since by Proposition 1.42 we are looking for a finite number of bounded integers, an algorithm
that comes to mind would be to simply try all possibilities. To do so, one can try all the finitely
many possibilities for χ until the characteristic equation χ(π) = 0 is satisfied in the Jacobian.
This amounts to a searchspace of size determined by the Weil bounds, at least in O(q g/2 ). The
complexity is therefore both exponential in g and log q.

The birthday-paradox approach

Replacing exhaustive search by a birthday-paradox approach, this running-time can be reduced
to the square root of the size of the search space, but there is also a memory cost of similar
magnitude. This was improved using distinguished points as in [99, 61], increasing the running
time by a constant factor but making the memory requirements negligible.
This approach is exponential in both log q and g, but it has the major advantage of being
massively parallelisable and having small memory requirements. This is the reason why, as we
will see in Sections 3 and 6, it is still used in practice to finish computations. It can also benefit
from previous knowledge on χ mod m as this reduces the size of the searchspace by a factor mg/2 .
12 Chapter 1. Point-counting and applications

Such information on χ can be gained using “polynomial-time” algorithms such as the ones we
describe below. For simplicity, we only present these methods in genus 2 and 3, respectively in
Chapters 3 and 6.

p-adic methods

Instead of considering the action of the Frobenius directly on JC , the p-adic approaches are
based on computing (a p-adic approximation of) a lift of the Frobenius and its action on some
differential forms. There are many algorithms following this philosophy, each using a different lift
or different differential forms. For instance, Satoh’s algorithm for elliptic curves [124] computes
the canonical lift of both the curve and the (dual) Frobenius endomorphism, whose action on the
lifted curve determines the trace of the Frobenius. Kedlaya’s algorithm [80] just needs a monic
lift but it acts on a larger space, namely a Monsky-Washnitzer cohomology group. Compared
to Satoh’s, this algorithm also has the advantage of working for hyperelliptic curves of arbitrary
genera, with a complexity in O(pg
e 4 n3 ) bit-operations and O(pg 3 n3 ) space. Note that Kedlaya’s

algorithm does not apply as such in characteristic 2, but this was fixed by Denef and Vercauteren
in [41]. This was extended by Tuitman in [142] for (possibly non-hyperelliptic) curves with a
“good” lift, where good hides various technical hypotheses that are expected to be satisfied in
general.
All these approaches have polynomial complexities in both g and n, and despite an improve-
√
ment by Harvey [69] reducing the dependency in p to p, this is still exponential in log p, which
is why they are used for fields of small characteristic. However, when counting points over many
fields, it is remarkable that an average polynomial-time complexity can be reached [71]. Indeed,
given a curve over Q, this algorithm computes the zeta function of its reduction modulo p on Fp
for all the primes p of good reduction smaller than N in time O(N e log3 N ) and polynomial in g.
Thus, on average, counting points on each curve amounts to a polynomial complexity in log p,
n and g. However, we still do not know any algorithm that has polynomial-time complexity in
all these parameters for counting points on a single curve.

Schoof’s algorithm and its extensions

input : An elliptic curve E/Fq given by the equation y 2 = x3 + ax + b

output: #E(Fq )
L←0
`←3
√
while L ≤ 4 q do
Compute ψ`
Let R = Fq [X, Y ]/ ψ` (X), Y 2 − X 3 − aX − b


/* this is E[`] */
2 2
In R, compute F0 = (X , Y ) ⊕E [q mod `](X, Y ) and F1 = (X q , Y q )
q q

Store t` the unique element of Z /` Z such that F0 = [t` ]F1

L←L·`
` ← NextPrime(`)
end
By CRT find t such that t ≡ t` mod ` for all previous `.
return q − t + 1
Algorithm 2: Schoof’s algorithm
1.2. Point-counting 13

In [127], Schoof describes an algorithm to compute the zeta function of an elliptic curve,
which amounts to computing the trace of the associated Frobenius endomorphism. The idea
is to consider the action of the Frobenius on the `-torsion subgroup to recover χ mod ` for
sufficiently many ` and, using Proposition 1.42, to recover χ by CRT.
Proposition 1.44. Let C be a smooth projective curve over Fq and ` coprime to q, then the
restriction of π to the `-torsion subgroup JC [`] has χ mod ` for characteristic polynomial.
Note that in Algorithm 2, ⊕E denotes the group law of the elliptic curve, which may lead to a
division by zero in the algebra R. To avoid this problem, one can previously factor ψ` to perform
operations in fields. This is a costly solution and we prefer to follow the approach of [40] and let
the representations of elements of R evolve during the computations. In the unlikely event of a
“forbidden” division, we can split ψ` as a product of two factors and pursue the computations
in the algebras obtained by replacing ψ` by each of its factors, with no consequence on the
complexity since each factor has a smaller degree. In this thesis, we sometimes reuse this
method under the name of “D5 strategy”. Another important aspect in practice is that we can
modify the group law to avoid handling the ordinate, and work only in a univariate algebra.
While this does not change the asymptotic complexity, it greatly reduces the running time.
Let us analyze the cost of one iteration of Schoof’s algorithm for a fixed `. First, ψ` can
be obtained from the recurrence formulas on the ψi ’s. These formulas show that computing
ψ` amounts to computing 5 ψk ’s with k ' `/2, which yields an overall complexity in O(`log2 5 )
for computing Cantor’s `-division polynomials. The bottleneck is the computation of F0 which
requires O(log q) operations in R, each of them accounting for a bit-complexity in O(` e 2 log q)
since ψ` has degree (`2 − 1)/2. Likewise, computing F1 is feasible within O(log q) operations in
R, and recovering t` can be done by exhaustive search for ` additions in R.
Thus, for a fixed ` the loop costs O(` e 2 log q(` + log q)) bit operations. Using results of
analytic number theory such as [140, Cor. 10.1], one sees that we have to repeat the loop about
O(log q/ log log q) times and the largest ` to consider has size O(log q). This proves that the
5
complexity of Schoof’s algorithm is in O(log
e q).
Schoof’s algorithm was later improved by restricting to specific primes ` for which we can
test the characteristic equation of the Frobenius in a proper subgroup of E[`]. This amounts to
replacing ψ` by a factor of degree O(`) in the definition of R, reducing the cost of each operation
4
in R by a factor ` and therefore having an overall complexity in O(log e q). We do not discuss
these improvements further and refer to [128] for more information.
Schoof’s algorithm relies on theoretical results such as Weil’s conjecture and Proposition 1.44
which are still valid even in a much more general setting, and it was extended few years later by
Pila [114] who proposed an algorithm to count points on Abelian varieties with time-complexity
in O((log q)∆ ), where ∆ depends on the dimension g of the input Abelian variety A, and its
group law.
Although most of the theoretical background is still valid in this much more general context,
to compute the action of π on A[`] we need to find an explicit description A[`] as a 2g-dimensional
vector space, so that given e ∈ A[`] we can compute π(e). This is the most difficult part and it
constitutes the bottleneck of many if not all the `-adic point-counting algorithms appearing in
this thesis. Pila’s approach to the problem is to view A[`] as a zero-dimensional algebraic set,
after getting a description for the maps [n], with n ≤ `. Applying straightforwardly the Frobenius
map to an element would not give a polynomial-time algorithm, but using the description of
A[`] we can repeatedly square elements and reduce by the defining equations. The complexity
result follows by bounding the number of monomials appearing in these equations, and applying
various primitives such as ideal membership testing and monomial bases computations.
14 Chapter 1. Point-counting and applications

As in Schoof’s algorithm, the complexity is polynomial in log q but the exponent ∆ is actually
exponential in the dimension g of A, so that the overall complexity is doubly exponential in g.
The dependency in g of the exponent of log q has later been improved by [75] and [3].
In [75], Huang and Ierardi reduce the dependency in g of the exponent in the case of plane
curves to a polynomial in g. This is achieved by using another way of representing JC [`]: by
considering its elements as divisor classes and by using effective Riemann-Roch algorithms to
get a semi-algebraic description with size polynomial in ` and exponential in the degree of C.
An important obstacle to overcome is the presence of singular points.
In [3], Adleman and Huang extend the result of [75] to Abelian varieties, with a more precise
2
complexity bound in (log q)O(g log g) for hyperelliptic curves of genus g. This time, the low-degree
representation of the Frobenius is achieved through faster ad hoc algorithms on semi-algebraic
sets.

1.3 Applications of point-counting

In this section, we review some of the various applications of point-counting. While some of
them are based on slight variations of previous algorithms, others only need the output of point-
counting algorithms. Some applications involve designing curves with special properties that
are closely related to the number of points on the (Jacobian of the) curve. In this context, the
so-called CM-method can be used to create curves with a prescribed number of points, instead
of applying point-counting algorithms to random curves until we are satisfied with the result.
For further information on this approach, we refer to [6].

1.3.1 Cryptographic use

Definition 1.45. Let (G, ⊕) be a cyclic group of order M and P a generator of G. The discrete
logarithm problem (DLP) in the group G is the problem of recovering the integer n ≤ M from
the element nP = P|
⊕ ·{z
· · ⊕ P}.
n times

This problem has lead to cryptographic applications taking advantage of the fact that the
exponentiation P 7→ nP is a one-way function as long as the DLP is hard.
In [131], Shoup defined a concept √ of generic group and proved that in such a group, any
algorithm must perform at least Ω( M ) group operations in order to compute a discrete loga-
rithm. There are many models for black box groups in the literature for which similar results
were proven, such as [112], but we do not intend to review them all.
However, finding such generic groups in real life is not that easy: for instance if G = Z /M Z,
the DLP can be solved in polynomial time by computing an XGCD. In real-life cryptography,
G is either the multiplicative group of a finite field or (the Jacobian of) an elliptic curve. Note
that the DLP in finite fields is much easier than in a generic group, as the complexity to solve
it range from quasipolynomial to subexponential, depending on its characteristic.
Thus, surprisingly enough, Jacobians of curves of fixed genus are the only known examples
of groups in which there is still no classical subexponential algorithm to solve the DLP. Yet,
some subexponential algorithms exist when g grows asymptotically as fast as log q and some
attacks like in [58], though still exponential, reduced the hardness of the DLP in genus strictly
larger than 2, making genus 1 and 2 optimal in terms of keysize. For a more detailed survey on
the subject, we refer to [53].
1.3. Applications of point-counting 15

We emphasize that even if we consider a group G in which the DLP is hard, exponential
algorithms may still be succesful in practice, for instance if #G is small. The following technique
due to Pohlig and Hellman in [116] shows that considering G of large size is not sufficient since
the difficulty of the DLP is entailed to the largest prime factor of #G.
Let us assume that G has order N = ri=1 pei i , where the pi ’s are distinct primes. Let
Q

Pi = Ni P with Ni = N/pei i , then the subgroup Gi generated by Pi has order pei i , so that we can
solve the DLP in G by solving it in each Gi and using the Chinese remainder theorem. Thus,
the DLP in G is as hard as the DLP in the “hardest” Gi .
We can now assume that G has a prime-power order N = pe . Given Q = nP , we want to find
n. Since n < N , we decompose n in basis p as n = e−1 i
P
i=0 ni p . Multiplying this decomposition
e−1
by p , we get
e−1
X
e−1 e−1 e
p n=p n0 + p ni p i .
i=1

Now since Q = nP , we have pe−1 Q = npe−1 Pso that pe−1 Q = n0 pe−1 P . We can now recover
n0 by solving a DLP, but in a group of size p instead of pe . Once done, we do the same for n1
and so on by induction. Finally, the DLP in G is broken down into solving e DLPs in groups of
order p.
To sum up, if we only focus on finding the smallest groups achieving a fixed security level,
then we have to choose (Jacobians of) curves of genus 1 and 2. But then, we must find curves
such that #JC = χ(1) is prime (or actually almost prime for other cryptographic reasons).
Because of Weil’s bounds, we already know that our curves have to be defined over a large
field. Although no practical attack against curves over fields of small characteristic has been
published, standards seem to prefer curves defined over Fp or Fp2 with p a large prime, so that
`-adic methods are more adapted in this context.
For elliptic curves, Schoof’s algorithm and its improvements based on Elkies and Atkin’s
work [128] are efficient enough to allow choosing random curves, counting points on them and
retaining only those with an almost prime order. The same method was used in [62] to create a
secure genus-2 curve, as we will detail in Chapter 3, but it involves much heavier computations.

1.3.2 Extensions of the Sato -Tate conjecture

Contrary to cryptographic applications for which only the size of the Jacobian is needed, we can
use the fact that most of the point-counting algorithms actually compute the full zeta function of
the curve. This has been used to study how the zeta functions (or the characteristic polynomials
of the p-Frobenius) of a fixed curve C over Fp behaves when p varies. In genus 1, this was
predicted by the Sato -Tate conjecture in 1948 and proven by Clozel, Harris, Shepherd-Barron
and Taylor in [32, 67, 139].

Theorem 1.46. Let E be an elliptic curve over Q and tp the trace of the p-Frobenius of its
reduction modulo a prime of good reduction p. If E does not have complex multiplication,
√ then
√ 2
the normalized traces tp /2 p are equidistributed with respect to the measure 2dt/π 1 − t .

Note that the distribution of these quantities was also known since Deuring for curves with
complex multiplication. A natural question is to ask for generalization of this statement in
higher genera, both in the general case and in less likely cases analogous to the CM case in
genus 1.
√
It is conjectured that given a curve C, the normalized Weil polynomials Lp (t)/ p of its
reductions modulo primes of good reduction follow a distribution that matches that of the
16 Chapter 1. Point-counting and applications

characteristic polynomials of random matrices of a compact subgroup of USp(2g), called the

Sato -Tate group of C. We refer to [78] and [129] for more information on this subject.
In genus 2 and 3, this was investigated in [49] and [82]. Although the exponential generic
approaches such as detailed in [134] are still faster than the p-adic ones for the range of curves
and Jacobians studied, average polynomial-time algorithms perfectly fit such investigations.

1.3.3 Algorithmic applications

Shortly after Schoof’s algorithm, Lenstra used elliptic curves to tackle the problem of factor-
ing integers with the celebrated elliptic curve method (ECM). In this section, we detail two
examples where curves, and more precisely point-counting on curves, are involved in designing
deterministic algorithms for number theory or computer algebra. Although point counting is not
involved in ECM, the number of rational points of the chosen elliptic curve plays an important
role since it has to be smooth.

Primality proving
Given an integer N , we want an algorithm running in time polynomial in N that returns “yes”
if N is prime and “no” if not, with a small probability of giving a wrong answer. We present
two algorithms in which `-adic methods play a central role, but let us first give an introductory
example.
Assume that we can find another integer m < N such that m − 1 is coprime to N and such
that m(N −1)/2 ≡ 1 mod N , then if (N − 1)/2 is prime, N is prime as well and we repeat the
process until the primality of N has been reduced to a number which is known to be prime (for
instance any prime smaller than 100). Since it is quite easy to find a good m quickly by taking
random integers, this would yield a probabilistic polynomial-time algorithm. But it has a fatal
flaw: if (N − 1)/2 is not prime we cannot draw any conclusion on the primality of N .
To deal with this obstacle, Goldwasser and Kilian [65] reduced the primality of N to that of
another integer r which is roughly two times smaller than N but can be different from (N −1)/2.
This is achieved by considering a random elliptic curve E and computing m = #E(Z /N Z)
using Schoof’s algorithm. Then if m happens to be even, one can prove that the primality of
r = m/2 entails that of N . Provided that there are sufficiently many “good” integers m occurring
as cardinalities of random elliptic curves such that r is actually prime, this method achieves
polynomial complexity. √ Unfortunately,
√ this amounts to proving that there are sufficiently many
primes between N − N and N + N , but current knowledge on the distribution of primes is
not even sufficient to even prove that there is a single prime in that interval.
Adleman and Huang found a workaround in [4] by devising two extensions of the previous
algorithm, and combining them together. First, instead of only considering the case m = 2r,
they reduced the primality of N to r such that m = λr with λ a small prime. This yields an
algorithm terminating in polynomial time for integers smaller than x outside of a subset of size
bounded by x15/16 . The other extension is to consider genus 2 curves instead of elliptic curves.
Indeed, while there is still a polynomial-time analogue
√ of Schoof’s algorithm for counting points,
the Hasse-Weil interval has a size larger than N N . This “reduces” the primality of N to that
of a larger integer, which could be a flaw at first sight. But they actually proved that after
repeating this step three times at most, they obtained a candidate prime large enough for the
first variant of the Kilian-Goldwasser algorithm to return the correct answer in polynomial time.
Agrawal, Kayal and Saxena later proposed a deterministic polynomial-time algorithm for
primality proving. Although these algorithms answer a theoretical question, we also remark that
1.3. Applications of point-counting 17

using elliptic curves for probabilistic primality testing is also competitive in practice thanks to
work of Atkin and Morain [7, 109]. Indeed, a recent computation using ECPP gave a primality
certificate for 2116224 − 15905 in November 2017 by Peter Kaiser [76].

Deterministic factorisation of polynomials over finite fields

A recent paper by Poonen [117] highlights the potential of `-adic methods to design a polynomial-
time deterministic algorithm for factoring polynomials over finite fields. This is based on an
idea by Kayal using Schoof’s algorithm in the following way. Let us assume that we are given
P ∈ Fp [t] such that P = (t − r1 )(t − r2 ), and we want to recover the two factors of P . By a
result of Berlekamp [15], one can reduce the problem of factorization in Fq [t] to that of factoring
polynomials in Fq [t] with distincts roots all in Fp , and by induction on the degree of P , handling
the case P = (t − r1 )(t − r2 ) is sufficient to perform factorization of any polynomial over a finite
field.
Defining B = Fp [t]/P , we consider an elliptic curve E on B. Actually, E splits as a cartesian
product of two elliptic curves over Fp , which we denote E1 and E2 . Assuming #E1 (Fp ) 6=
#E2 (Fp ), the respective traces t1 and t2 of the Frobenius of E1 and E2 are also different.
Therefore, there is a prime ` such that t1 6≡ t2 mod `. When applying Schoof’s algorithm on E
as if B were a field, we end up considering that ` and looking for a candidate t0 for the trace of
the Frobenius of E modulo `. Doing so by exhaustive search we encounter special phenomena
for t0 = t1 and t0 = t2 , and eventually recover r1 and r2 . Indeed, for t0 = t1 , all the elements of
the curve E over Fp [t]/(t − r1 ) satisfy the characteristic equations φ2p − t0 φp + p = 0, but not all
the elements of E, since t1 6= t2 . This leads to a division by a non-invertible element in B, itself
leading to a proper factor of P .
If #E1 (Fp ) = #E2 (Fp ), however, this does not work so we have to choose another E and
hope not to fall in the same pathologic case. It is reasonable to think that there is enough room
for the choice of E to end up in a good situation after only a few attempts, but this is still
unproved. To increase the chances of success, Poonen suggests to switch to higher-dimensional
Abelian varieties and use Pila’s algorithm instead of Schoof’s, as their zeta functions have g
degrees of freedom instead of one. Although it is even more convincing, the fact that we have
“enough” different zeta functions remains unproved.

Other applications
Schoof’s algorithm and its generalization all rely on having a nice representation of the `-torsion,
in a sense that we have already mentioned, and will make clearer in Chapters 3 to 5. An example
is given in [122, Sec. 7.5] to compute all the `-isogenies from an Abelian variety knowing its `-
torsion subgroups.
Last, some multiplication algorithms like [31] or algebro-geometric codes benefit from curves
with many rational points [66, 107, 79]. We do not further develop these aspects since they use
mostly non-hyperelliptic curves.
18 Chapter 1. Point-counting and applications
 Chapter 2

Polynomial systems

The generalizations of Schoof’s algorithm all rely on describing the `-torsion in a way that
allows to test ideal membership and perform group operations. In genus greater than 1, this
step is the bottleneck of these algorithms, and therefore the step to improve in order to get
better complexity estimates. The direction that we investigate in this thesis consists in formally
multiplying a divisor D by ` and then solving the polynomial system obtained after equating
`D = 0. The aim of this section is to define what we mean by polynomial system solving, to
present the methods that we use to do so and to study their complexities. These methods and
complexity results are used in Chapters 3 to 6.
Since all our systems will be designed to model (subsets of) the `-torsion of Abelian varieties,
they will all have dimension zero. Thus, all the definitions and statements of this section are
given in the particular case of zero-dimensional systems.
In this chapter, we review three methods for solving polynomial systems along with com-
plexity results that we reuse later. Section 2.2 recalls algorithms for computing Gröbner bases,
but their complexities are hard to bound, so that they are only used for practical results in
Chapter 6. Section 2.3 deals with resultants that provide a good alternative both in theory and
in practice for bivariate systems, as detailed in Chapter 3. In the trivariate case, they are no
longer competitive against algorithms like F4 but they can still be used to derive complexity
bounds in Chapter 6. Lastly, Section 2.4 is dedicated to the geometric resolution, a method
used in Chapter 5 to take advantage of structural properties of our polynomial systems.

2.1 Solving polynomial systems

Definition 2.1. Let K be a field, and let f1 , . . . , fm be polynomials in K[x1 , . . . , xn ]. The
solutions of the polynomial system {f1 , . . . , fm } are the tuples (z1 , . . . , zn ) ∈ K̄ n such that for
all i ∈ {1, . . . , m}, fi (z1 , . . . , zn ) = 0. When the set of solutions is finite, we say that the system
is zero-dimensional (or has dimension zero). In that case, we refer to the number of solutions
(in K̄) counted with multiplicities as the degree of the system.
The simplest possible case of operations in a quotient ring is the univariate case. For instance,
given an elliptic curve E the `-division polynomials allow us to reduce the computations of the
Frobenius in E[`] to exponentiation in the quotient ring Fq [X]/ψ` (X). In more general cases, we
always fall back to the univariate case using one of the following strategies. Either we eliminate
variables one by one to end up with one univariate equation, or we parametrize all the variables
by another one. More precisely, we say that we have solved a system when we have put it in
one of the two following forms.

19
20 Chapter 2. Polynomial systems

Definition 2.2 (Triangular form). We say that a zero-dimensional polynomial system is trian-
gular if it has the form
g1 (x1 , x2 , x3 , . . . , xn )
..
.
gi1 (x1 , x2 , x3 , . . . , xn )
gi1 +1 (x2 , x3 , . . . , xn )
..
.
gi2 (x2 , x3 , . . . , xn )
gi2 +1 (x3 , · · · , xn )
..
.
gin (xn )

Actually, we can often get an even simpler form like

x1 − h1 (xn )
..
.
xn−1 − hn−1 (xn )
hn (xn ).

When a system can be put in this form, we say that the system is in shape position. It has been
proven in [14] that when the associated ideal is radical this is very likely after a random linear
change of variables, provided that the field of definition is large enough.

Definition 2.3 (Geometric resolution). A geometric resolution of a zero-dimensional polynomial

system is a linear combination x0 of the variables xi ’s and a system of the form

h0 (x0 ) = 0
x1 = h1 (x0 )
..
.
xn = hn (x0 )

where h0 is a univariate polynomial whose degree D is the degree of the polynomial system, and
the hi are univariate polynomials of degrees smaller than D. The linear combination x0 is called
a separating variable or a primitive element.

To compute a triangular form of our system, a possible strategy is to eliminate one variable
and then repeat the same procedure on the equations with n − 1 variables. We made this precise
by introducing the following definition

Definition 2.4. [38, Sec. 3 Def. 1] Given I = hf1 , . . . fm i ⊂ K[x1 , . . . , xn ], the k-th elimination
ideal Ik is defined by
Ik = I ∩ K[xk+1 , . . . , xn ].

An elimination scheme is an algorithm to compute a generating set of Ik , or at least a set of

elements of Ik . A historical example of elimination is the Gauß-Jordan elimination for solving
2.2. Gröbner bases 21

linear systems of equations. This method can be seen as computing the row-reduced form of a
matrix associated to the system, so it is no surprise that we end up with a system in triangular
form. In Sections 2.2 and 2.3, we review two ways of performing elimination, respectively by
computing a Gröbner basis or resultants.

2.2 Gröbner bases

This section presents properties of Gröbner bases and explains why they are a particularly
convenient tool for solving polynomial systems. We briefly present known strategies to compute
them and review complexity results.

2.2.1 Gröbner bases and elimination

Definition 2.5. A monomial xα in K[x1 , . . . , xn ] is an element xα1 1 . . . xαnn with α = (α1 , . . . , αn )
a tuple of nonnegative integers. The total degree of such a monomial is the sum |α| = ni=1 αi .
P

Definition 2.6. A monomial ordering on K[x1 , . . . , xn ] is a relation ≺ on Zn≥0 such that

• if α ≺ β and γ ∈ Zn≥0 , then α + γ ≺ β + γ,

• ≺ is a well-ordering, i.e. it is a strict total ordering such that every nonempty subset of
Zn≥0 has a smallest element under ≺.
This gives an ordering on the set of monomials via α 7→ xα which is compatible with the
multiplication of monomials. Given an element P = α aα xα of K[x1 , . . . , xn ], it allows to
P

define:
• the multidegree mdeg(P ), the greatest α (for the monomial order) such that aα 6= 0,

• the leading monomial LM(P ) = xmdeg(P ) , the greatest monomial appearing in P ,

• the leading coefficient LC(P ) = amdeg(P ) , the coefficient of the leading monomial of P ,

• the leading term LT(P ) = LC(P ) LM(P ),

If S ∈ K[x1 , . . . , xn ] is a set of polynomials, we define LT(S) = {LT(P ) | P ∈ S}.
In this thesis, we mostly encounter the following two monomial orderings:
Example 2.7. (Lexicographic order). Let α and β be two elements in Zn≥0 , we write α ≺lex β
if there exists i such that αj = βj for any j < i and αi < βi .
In other words, the monomials are ordered by lexicographic order using the order xn ≺ xn−1 ≺
· · · ≺ x1 for the variables.
Example 2.8. (Graded reverse lex order). Let α and β be two elements in Zn≥0 , we write
α ≺grevlex β if |α| < |β| or if |α| = |β| and there exists i such that αj = βj for any j > i and
αi > βi .
In other words, grevlex orders first by total degree and then uses the reverse lexicographic
order to compare in case of equality, using the order x1 ≺ x2 ≺ · · · ≺ xn for the variables.
Consider the case of a system of two univariate polynomials, {P (X), Q(X)}. A triangular
form of this system is {gcd(P (X), Q(X))}, and the GCD computation can be done using Euclid’s
algorithm, i.e. successively reducing one polynomial by the other. In what follows, we introduce
definitions to extend the notion of reduction to the multivariate case.
22 Chapter 2. Polynomial systems

Theorem 2.9. [38, Chap. 2, Th. 3] Let us fix a monomial order and let F = (f1 , . . . , fs ) be an
ordered tuple of polynomials in K[x1 , . . . , xn ]. Every f ∈ K[x1 , . . . , xn ] can be written
s
X
f= ai fi + g,
i=1

with the ai ’s and g in K[x1 , . . . , xn ] such that g is either 0 or a linear combination of monomials
that are not divisible by any of the LM (fi )’s. Furthermore, if ai fi 6= 0, then we have mdeg(f ) ≥
mdeg(ai fi ).
F
Definition 2.10. In the setting of the previous theorem, we denote g = f and call it a re-
mainder of f modulo F . Furthermore, if LM(g) ≺ LM(f ), we say that f is top-reducible.

These definitions generalize the univariate Euclidean division but in a much weaker sense:
even given a fixed monomial ordering, there is no unicity of the remainder in general. For
some well-chosen sets F , however, the remainder modulo F is unique and it is thus possible to
perform an analogue of Euclid’s algorithm. In the next section, we define Gröbner bases that
are an example of such nice sets.

Definition 2.11 (Gröbner basis). Let I be an ideal of K[x1 , . . . , xn ], ≺ a monomial ordering and
a finite subset G ⊂ I. Then G is a Gröbner basis of I for the order ≺ if hLM (G)i = hLM (I)i.

Theorem 2.12. [38, Chap. 2, §5 Cor. 6] Given a monomial ordering ≺, any nonzero ideal has
a Gröbner basis for ≺.

The previous theorem guarantees the existence of a Gröbner basis but there is no unicity:
given G a Gröbner basis, the set G0 obtained by adding any element in I is another Gröbner basis.
In the following definition, this inconvenience is fixed by adding some minimality condition.

Definition 2.13. A Gröbner basis G of I is said to be reduced if for all h ∈ G we have LC(h) = 1
and no monomial of h is in hLM (G\{h})i.

Proposition 2.14. [38, Chap. 2, §7 Prop. 6] Let I be a non-zero ideal of K[x1 , . . . , xn ] and ≺
a monomial ordering. Then I has a unique reduced Gröbner basis G for ≺.

Note that while the reduced Gröbner basis of I for a monomial order is unique, it may differ
from the reduced Gröbner basis for a different monomial order. As announced previously, one
of their essential features is the unicity of the reduction of a polynomial by a Gröbner basis, as
defined in Definition 2.10.

Proposition 2.15. Let G = {g1 , . . . , gk } be a Gröbner basis of an ideal I and let f ∈ K[x1 , . . . xn ].
Then there exists a unique r ∈ K[x1 , . . . xn ] such that:

• no monomial of r is divisible by any LT(gi ), i.e. r is in normal form modulo G,

• there exists h ∈ I such that f = h + r.

G
The unique r is called the normal form of f modulo G, still denoted r = f .

The following proposition gives additional characterizations of Gröbner bases.

Proposition 2.16. A finite set G is a Gröbner basis of an ideal I if one of the following
equivalent properties is satisfied:
2.2. Gröbner bases 23

• for every f ∈ I, at least one of the reductions of f modulo G is zero.

• every non-zero f ∈ I is top-reducible modulo G,

• for every f ∈ I, there exists g ∈ G such that LM(g) divides LM(f ).

Theorem 2.17 (The elimination theorem). [38, Sec. 2, Th. 2] Let I ⊂ K[x1 , . . . , xn ] be an ideal
and let G be a Gröbner basis of I with respect to the lexicographic order where xn ≺ . . . ≺ x1 ,
then for every j ≤ n the set
Gj = G ∩ K[xj+1 , . . . , xn ]
is a Gröbner basis of the j-th elimination ideal Ij = I ∩ K[xj+1 , . . . , xn ].
In particular, this shows that a Gröbner basis of an ideal for the lexicographic order is in
triangular form.

2.2.2 Computing Gröbner bases

The first algorithm to compute Gröbner bases was introduced by Buchberger in 1965. Like Gaus-
sian elimination, it relies on cancelling the leading monomials of two polynomials by combining
them.
Definition 2.18. Let P and Q be two polynomials in K[x1 , . . . , xn ], the S-polynomial of P and
Q with respect to the monomial ordering ≺ is the combination
lcm(LM(P ), LM(Q)) lcm(LM(P ), LM(Q))
S(P, Q) = P− Q.
LT(P ) LT(Q)
Then LM(S(P, Q)) is strictly smaller (for ≺) than lcm(LM(P ), LM(Q))). We call (P, Q) a
critical pair and S(P, Q) the S-polynomial associated to the critical pair.
This gives another characterization of Gröbner bases.
Proposition 2.19. Let G = {g1 , . . . , gk } be a subset of K[x1 , . . . , xn ] not containing 0 and let
Sij = S(gi , gj ) be the S-polynomials for the monomial ordering ≺. Then G is a Gröbner basis
of hg1 , . . . , gk i if and only if for any i, j, at least one of the reductions modulo G of Sij is zero.
Buchberger’s algorithm constructs Gröbner bases by forcing this proposition to be satisfied:
starting from a set F = {f1 , . . . , fm }, compute all the S-polynomials, reduce them by F and
repeat the operation to the union of F and all the non-zero remainders modulo F .
Note that we still have to worry about the termination of Buchberger’s algorithm. Dickson’s
lemma states that any monomial ideal has a finite basis, which is equivalent to the fact that
there is no infinite increasing sequence of monomial ideals. See for instance [38, Chap. 2, Sec. 4
& 5] for statements and proofs.
There is an extensive literature on improvements to Buchberger’s algorithm, but we mainly
focus on two types of improvement that we used in practice in Chapter 6. The first idea is
to compute Gröbner bases for the grevlex order as their computations involve polynomials of
smaller degrees. This is noticeable both in practice and in the complexity bounds given in [91].
However, grevlex bases are often not sufficient to directly solve a system, contrary to their lex
counterparts. We can circumvent this difficulty by a change of ordering using either the FGLM
algorithm [47] in dimension zero, or a Gröbner walk [35] in positive dimension.
Since most of the running time of Buchberger’s algorithm is spent computing reductions of
critical pairs, the choice of the order in which we reduce them plays a prominent role. There is
24 Chapter 2. Polynomial systems

input : F = {f1 , . . . , fm } and ≺ a monomial ordering

output: A Gröbner basis G for ≺
G ← F , G0 ← ∅
while G 6= G0 do
G0 ← G
for each critical pair (P, Q) with P, Q in G0 and P 6= Q do
G0
S ← S(P, Q)
if S 6= 0 then
G ← G ∪ {S}
end
end
end
return G
Algorithm 3: Buchberger’s algorithm as in [38, Chap. 2, Th. 2].

no absolute answer to this question, but practical experiments allow to compare the efficiency of
different choices. The so-called normal strategy consists of reducing first by pairs of small degrees
and seems to be quite efficient. Another improvement was brought by the F4 algorithm [45],
using linear algebra to perform reductions much faster. The link between Gröbner bases and
linear algebra will be detailed in the next section, as it is also helpful to prove complexity bounds.
Further improvements on the reduction step can be designed, for instance by anticipating
and avoiding reductions of some critical pairs to zero. This idea was introduced by Buchberger’s
criteria (see [38, Sec. 9]) and later improved in the F5 algorithm [46].

2.2.3 Complexity results

Let us consider an ideal I generated by m homogeneous polynomials fi ∈ K[x1 , . . . , xn ] of
respective degrees di . We introduce Id = {f ∈ I | deg f = d} and point out that it is a vector
space of finite dimension. Since any element
of Id can be decomposed in the basis of degree-d
monomials of K[x1 , . . . , xn ], dim Id ≤ n+d−1
d .
Definition 2.20. Let I = hf1 , . . . , fm i be a homogeneous ideal. A finite set G is a d-Gröbner
basis of I if it generates I and if any of the following equivalent statements hold:
G
• ∀g1 , g2 ∈ G, S(g1 , g2 ) = 0 as long as deg S(g1 , g2 ) ≤ d,
• every f ∈ I with deg(f ) ≤ d is top-reducible by G.
Note that a d-Gröbner basis is also a k-Gröbner basis for k ≤ d, so that a sequence Gi of i-
Gröbner bases is increasing. Thus, the ascending chain condition implies that Gi is stationnary,
i.e. there is a D such that Gk = GD for any k ≥ D. Hence, for k ≥ D, a k-Gröbner basis is
a Gröbner basis. An algorithm computing d-Gröbner basis can be derived from Buchberger’s
algorithm by only considering S-polynomials of degrees ≤ d.
To make the link between Gröbner bases and Gaussian elimination even clearer, one can
represent the vector space Id by a matrix whose columns are indexed by the degree-d monomials
of K[x1 , . . . , xn ] (in decreasing order for ≺) and the rows by the degree-d multiples of the m
generators of I. This was introduced by Macaulay in [97] and the matrix is named the degree-
d Macaulay matrix of I. It enables to perform operations in I by using linear algebra, and
generalizes the notion of Sylvester matrix (see Definition 2.30, below).
2.2. Gröbner bases 25

Definition 2.21. A homogeneous polynomial f of degree d is generic if it can be written as

Ui1 ,...,in xi11 · · · xinn ,
X
f=
i1 +···+in =d

where the Ui1 ,...,in are variables.

Definition 2.22. Let f1 , . . . fn be generic homogeneous polynomials of respective degrees di in
the variables x1 , . . . , xn such that the variables appearing as coefficients of the fi are all distinct.
The Macaulay resultant of the fi ’s is the GCD of all the minors of maximal size of the degree-d
Macaulay matrix, with d = ni=1 (di − 1) + 1.
P

Proposition 2.23. [97] The Macaulay resultant R is a homogeneous polynomial, irreducible

and of degree Di = j6=i dj in the coefficients of fi . The system {f1 , . . . , fn } has a non-trivial
Q

solution if and only if R vanishes.

In [91, 92], Lazard performs Gaussian elimination to the degree-d Macaulay matrices for
d ≤ D. This yields a D-Gröbner basis, which is a Gröbner basis for D large enough. From
the maximal degree of polynomials appearing in the computations and the size of the Macaulay
matrix of that degree, one can deduce a complexity bound for Lazard’s algorithm. However, as
in Buchberger’s algorithm, we expect this algorithm to perform many unnecessary reductions
to zero since the matrix has a rank much smaller than its size. Using the F5 criteria, it is
possible to consider a much smaller matrix and design a matrix-based counterpart to the F5
algorithm [46].
To construct the degree-d Macaulay matrix in the affine case, the columns are indexed by all
the monomials of degree ≤ d and the rows by multiples of the fi ’s by monomials such that the
product has degree ≤ d. As previously, a d-Gröbner basis is computed by reducing the affine
degree-d Macaulay matrix.
A similar method to find a solution of a polynomial system is to solve the linear system
M X = 0 with M the Macaulay matrix of the system. We do not detail it further and refer
to [36] for the introduction of XL and its application to cryptanalysis.
In the worst case, the cost of computing a Gröbner basis is doubly exponential in the number
of variables (see [100]), but this bound was reached using tailored systems and is pessimistic even
for random inputs. In fact, most systems that we encounter have particularities that make them
even simpler to solve. Indeed, we seldom want to solve random polynomial systems but rather
focus on examples coming from specific contexts. In our case, the ideals that we consider are
zero-dimensional by nature, a special case for which the complexity drops to simply exponential.
In the overdetermined case, the following theorem states that the maximal degree is at least
twice smaller than the Macaulay bound.
Theorem 2.24. [136] Let f1 , . . . , fn+1 be a generic system (i.e. the coefficients of the fi ’s
are parameters) of respective degrees d1 , . . . , dn+1 in K[x1 , . . . , xn ]. Then the Macaulay resul-
tant
P of the homogeneized
 system can be computed from a Macaulay matrix of degree at most
n+1
i=1 (di − 1) + 1 /2.
The Macaulay bound gives a dependency of the complexity in the arithmetic mean of the
degrees of the equations. On the other hand, one can wish for results involving their geometric
mean given by the following theorem due to Lakshman and Lazard.
Theorem 2.25. [89, Th. 1] There exists a probabilistic algorithm which, given a zero-dimensional
system, computes a Gröbner basis of its radical over the field of coefficients in time polynomial
in the Bézout bound i = 1n di .
Q
 26 Chapter 2. Polynomial systems

The remainder of the section presents a tighter complexity estimate from [11], when further
assumptions are made on the system. These assumptions guarantee that all the trivial reductions
in Buchberger’s algorithm are avoided using the F5 criteria.

Definition 2.26 (Regular sequence). [11, Def. 1.7.1] Let f1 , . . . , fm be a sequence of homoge-
neous polynomials in K[x1 , . . . , xn ]. We say that the sequence is regular if the following condi-
tions hold

• hf1 , . . . , fm i =
6 K[x1 , . . . , xn ]

• for 2 ≤ i ≤ m, if gi fi ∈ hf1 , . . . , fi−1 i, then gi ∈ hf1 , . . . , fi−1 i.

Definition 2.27. [11, Def. 1.7.2] Let f1 , . . . , fm be a sequence of polynomials in K[x1 , . . . , xn ].

Denote by fih the homogeneous part of highest degree in fi . We say that the sequence f1 , . . . , fm
is regular if the sequence of homogeneous polynomials f1h , . . . , fm
h is regular.

Definition 2.28 (Noether position). A homogeneous ideal I of K[x1 , . . . , xn ] is in Noether

position if there exists r ≤ n such that I ∩ K[x1 , . . . , xr ] = (0) and K[x1 , . . . , xn ]/I is an integral
extension of K[x1 , . . . , xr ].

Let f1 , . . . , fm be a sequence of elements in K[x1 , . . . , xn ] with m ≤ n and di the degree of fi .

Let us fix the monomial ordering grevlex with xn ≺ · · · ≺ x1 and make the following hypotheses:

(Hyp. 1) the sequence f1 , . . . , fm is regular

(Hyp. 2) for any 1 ≤ i ≤ m, the ideal hf1 , . . . , fi i is in Noether position.

Define gd,i (n) as the coefficient of z d in the expansion of

i−1
z di Y
(1 − z dk ).
(1 − z)i−1 k=1

The gd,i (n) bound the number of degree-d polynomials in the Gröbner basis of hf1 , . . . , fi i for
the grevlex ordering [11, Th. 3.4.1] . The previous expression is in fact a polynomial whose
degree equals the Macaulay bound ij=1 (dj − 1) + 1.
P

Theorem 2.29. [11, Th. 3.4.2] Under Hypotheses 1 and 2, there exists an algorithm to com-
pute the Gröbner basis of hf1 , . . . , fm i which performs a total number of elementary operations
bounded by
m−1 ∞
! !
X X i + d + di+1 n + d + di+1 − 1
gd+di+1 ,i+1 (n) .
i=1 d=0
d + di+1 d + di+1

This result is achieved using a variant of the F5 algorithm, and although it is not easily
compared to other complexity bounds, it is instantiated in particular cases in [11, 12] in which
a simpler complexity bound is derived and indeed yields an improvement over that of Lazard’s
algorithm. As explained in [11], these bounds where notably used for solving equations over F2
and in particular attacking the public-key system HFE, as well as decoding codes. In our setting
however, the regularity hypotheses fail and the complexity bound is not tight enough, just as
the Bézout bound.
Although Gröbner bases are a powerful tool for polynomial system solving, we cannot use
them to derive asymptotic complexity estimates because only the most general and pessimistic
2.3. Resultant-based approaches 27

complexity bounds apply to our setting. In particular, the Bézout bound is much too large for our
purpose as it does not take into account the fact that most variables come with a small degree,
contrary to its multihomogeneous counterpart. In practice however, we will see in Chapter 6
that they are particularly efficient to solve trivariate systems. The next two sections review
alternative options for which we have tighter complexity estimates, and Section 2.4 presents a
method to achieve a polynomial-time complexity in the multihomogeneous Bézout bound, which
is the cornerstone of Chapter 5.

2.3 Resultant-based approaches

2.3.1 Resultants and elimination
Definition 2.30. Let m and n be two positive integers, and P = a0 X m + · · · + am and Q =
b0 X n + · · · + bn be two polynomials in K[X]. We define Syl(P, Q) the Sylvester matrix of P and
Q as  
a0 a1 · · · am

 a0 a1 · · · am 


.. .. 

 . ··· . 

 
 a0 a1 ··· am 
.
 
b
 0 b1 ··· bn 

 b0 b1 ··· bn 


.. .. 

 . ··· . 

b0 b1 ··· bn
Definition 2.31. The resultant of P and Q, denoted Res(P, Q) or ResX (P, Q) is the determi-
nant of the Sylvester matrix defined above.
There is a strong link between resultants and GCD, as the last non-zero row of the row-
echelon form of Syl(P, Q) contains the coefficients of a GCD of P and Q. In particular, the
degree of GCD(P, Q) is the corank of Syl(P, Q) and we have the following result:
Proposition 2.32. The polynomials P and Q are coprime if and only if Res(P, Q) 6= 0.
There is an extensive literature on the numerous properties of the resultant, but we only
detail those we will reuse. For more information on the subject, we refer for instance to [5].
Let us now remark that it is possible to define resultants even when the coefficient ring is not
a field, and in particular when P and Q live in R[X] with R = K[Y ]. In that case, ResX (P, Q)
is a polynomial in Y but no longer in X and we expect that the previous proposition still holds,
i.e. that the solutions of P (X, Y ) = Q(X, Y ) = 0 satisfy ResX (P, Q)(Y ) = 0. If so, by adding
either the equation P (X, Y ) = 0 or Q(X, Y ) = 0 we have put the system in triangular form.
However, the following proposition shows that those two systems are not equivalent since we
may have added additional solutions.
Proposition 2.33 (Prop 6.4, [21]). Let m and n be two positive integers, P = a0 X m + · · · + am
and Q = b0 X n + · · · + bn be two polynomials in (K[Y ])[X], with K an algebraically closed
field. Then the roots of ResX (P, Q) ∈ K[Y ] are the y-coordinates of the solutions of the system
P = Q = 0 and the common roots of the leading coefficients a0 (Y ) and b0 (Y ).
In our bivariate example, we eliminated a variable using a resultant and then carried on
one of the initial equations to have a triangular system. In many cases and in particular in
28 Chapter 2. Polynomial systems

the one encountered in Chapter 3, the additional equation can be “nicer”, i.e. of the form
S1 (Y )X − S0 (Y ). Such an equation is given by one of the subresultants defined below.

Definition 2.34 (Subresultant matrix [141]). Let m, n and j be three positive integers, and
P = a0 X m + · · · + am and Q = b0 X n + · · · + bn be two polynomials in K[X]. We define the
j-th subresultant matrix of P and Q as the (n + m − 2j) × (n + m − j) submatrix N (j) (P, Q) of
the Sylvester matrix by taking the top m − j rows of coefficients of P and the top n − j rows of
coefficients of Q.
  
a0 a1 · · · an 



 a0 a1 ··· an 




.. .. ..  m−j

 . . . 
 


  
 a0 ··· an  
   .
b
 0 b1 ··· bm 
 



 b0 b1 ··· bm 




.. .. ..  n−j

 . . . 
 



b0 ··· bm 

Definition 2.35 (Subresultant [141]). Keeping the notation of the previous definition, for k ≤ j,
(j)
we further define Nk (P, Q) as the (square) submatrix of N (j) (P, Q) obtained by taking only its
rightmost m + n − 2k − 1 columns and its (m + n − j − k)-th column. The j-th subresultant of
P and Q is then defined as the polynomial
j
X (j)
det Nk X k .
k=0

Considering P and Q in K[X, Y ], one can define the bivariate resultants and subresultants
except that the coefficients ai and bj are now polynomials in (say) Y . Thus, the bivariate
resultant R(Y ) = ResX (P, Q) is now a univariate polynomial and the j-th subresultant is a
bivariate polynomial of degree at most j in X. If the first subresultant of P and Q is non-
zero, then it has degree 1 in X so that we can write it S1 (Y )X + S0 (Y ) with S1 a non-zero
univariate polynomial. Since the resultant and subresultants of P and Q are all in the ideal
generated by P and Q, the system P (X, Y ) = Q(X, Y ) = 0 is equivalent to the system {R(Y ) =
0, S1 (Y )X + S0 (Y ) = 0}.
Let us consider P and Q in K[x1 , . . . , xn ], which we view as R[X] = K[x2 , . . . , xn ][X]. We
can likewise define the Sylvester matrix and resultant Resx1 (P, Q) ∈ K[x2 , . . . , xn ]. As in the
bivariate case, we eliminate the variable x1 but the resultant is not necessarily a generator of
the first elimination ideal. However, we will see in Chapter 6 that successive elimination by
resultants is still accurate enough for us to use it in the trivariate case with an asymptotic
complexity that matches that of more sophisticated methods. In the remainder of the section,
we give more details about the complexity of computing resultants of polynomials in up to three
variables.

2.3.2 Computing univariate resultants

Consider P and Q two univariate polynomials over K. To compute Res(P, Q), an algorithm
that comes in mind would be computing the determinant of the Sylvester matrix. This can be
2.3. Resultant-based approaches 29

done in O(nω ), where n is a bound on the degrees of P and Q and ω < 2.38 the exponent of
linear algebra. Using the fact that Res(P, Q) = (−1)mn bm−r 0 Res(Q, R) with R the remainder
of the Euclidean division of P by Q, one can design a algorithm that returns Res(P, Q) in time
quadratic in n. The subresultants can similarly be related to (variations of) Euclid’s algorithm
by the fundamental theorem of subresultants (see for instance [141, Th. 3.4.]).
In general, by following Euclid’s algorithm, we have a sequence of polynomials whose degrees
decrease by one at each step so that n steps are needed and the complexity is indeed quadratic.
However, one can imitate the half-GCD algorithm to halve the degree at each step. This yields
a quasi-optimal algorithm for computing the resultant and the last non-zero subresultant of two
univariate polynomials P and Q.
Properly presenting a fast algorithm for computing the resultant of two polynomials along
with their last non-zero subresultant is not a challenge that we want to take in this thesis,
insomuch as we did not contribute on this aspect. We therefore limit ourselves to stating
the following theorem, which is the only statement about (sub)resultants that will be needed
throughout this thesis.

Theorem 2.36 (Computing resultants and subresultants). [21, Prop. 6.15 & Thm. 6.16] Let
P and Q be two univariate polynomials in K[X] of degrees bounded by n > 0. Then Res(P, Q)
can be computed in time and space O(n),
e and so can any subresultant of P and Q.

For our purpose, we will also need to compute bivariate and trivariate (sub)resultants, for
which the existence of a quasi-optimal algorithm is still an open problem. In the next section,
we give complexity bounds for computing these resultants by using evaluation / interpolation
schemes to reduce to the univariate case.

2.3.3 Bivariate and trivariate resultants

In Chapters 3 and 6, we put the equations of the `-torsion ideal in triangular form by successively
eliminating variables using resultants. In this section, we bound the complexity of computing
bivariate and trivariate resultants. We also provide bounds on the degrees of the resultants,
either because they intervene in another complexity result or for the following reason: we have
seen that the resultant of two polynomials belongs to the elimination ideal but there is no
guarantee that it is a generator, and it can even be zero. This gives rise to extraneous solutions
of our system that we are not interested in and that we call parasites. When the resultants are
not zero, bounding their degrees is a way of controlling the number of parasites. In our case,
we can use this to ensure that parasites do not harm the asymptotic complexity. On the other
hand, they are not innocuous in practice and part of Chapter 3 is dedicated to reducing their
number.

Definition 2.37 (Evaluation-Interpolation). Given n distinct elements a0 , . . . , an−1 in a field

K, and P ∈ K[X] a polynomial of degree < n, we call (multipoint) evaluation the computation
of P (a0 ), . . . , P (an−1 ).
Conversely, given b0 , . . ., bn−1 , n additional elements of K we call interpolation the compu-
tation of a polynomial P of degree < n such that P (a0 ) = b0 , . . . , P (an−1 ) = bn−1 .

Theorem 2.38. [21, Th. 5.1] Given n distinct field elements a0 , . . . , an−1 , one can perform the
multipoint evaluation or the interpolation in O(n)
e field operations.

Note that when K = Fq , we may not have enough distinct points to perform evaluation or
interpolation of a polynomial of large degree. However, when it is the case, we can take a field
30 Chapter 2. Polynomial systems

extension Fqδ of Fq , and that will add a factor O(δ)

e to the complexity. The complexity of the
algorithms will be polynomial in the number of evaluation points, therefore, the final complexity
will be logarithmic in δ, so that the cost of taking a field extension will be hidden in the O()
e
notation. We will therefore not mention this potential complication further.
Another difficulty is that an evaluation / interpolation strategy assumes that the points of
evaluation are generic enough, so that all the degrees after evaluation are generic. This is again
guaranteed by taking a large enough base field. Still, the algorithm remains a Monte-Carlo one.
In Chapters 3 and 6, the final results of our algorithms can readily be tested, which is why they
are Las Vegas even though they involve resultants-based primitives that are not.

Proposition 2.39. [54, Thm. 6.22 and Cor. 11.21] Let P (x, y) and Q(x, y) be two polynomials
whose degrees in x and y are bounded by dx and dy respectively. Then, R(y) = Resx (P, Q) can
e 2 dy ) field operations, and the degree of R is bounded by 2dx dy .
be computed in O(d x

Proposition 2.40. Let P (x, y, z) and Q(x, y, z) be two polynomials whose degrees in each vari-
e 5 ) field operations,
able are bounded by d. Then, R(y, z) = Resx (P, Q) can be computed in O(d
2
and the degree of R in each variable is bounded by 2d .

Proof. The Sylvester matrix has at most 2d columns and its entries are bivariate polynomials
whose degrees in y and z are bounded by d. Thus, its determinant is a polynomial whose degrees
in y and z are bounded by 2d2 .
2
We first perform a Kronecker substitution by considering P̃ (x, y) = P (x, y, y 2d +1 ) and
2
Q̃(x, y) = Q(x, y, y 2d +1 ), which are polynomials of degrees ≤ d in x and ≤ 2d3 + d in y. Note
2
that the choice to replace z by y 2d +1 is made to be able to invert the Kronecker substitution
after the resultant computation.
Next, we compute R̃(y) = Resx (P̃ (x, y), Q̃(x, y)). By Lemma 2.39, it is a univariate polyno-
mial of degree at most 4d4 + 2d2 and can be computed in O(d e 5 ) operations. We can then invert
the Kronecker substitution to get R(y, z), which can be done in time linear in the number of
monomials, that is in O(d4 ).

Proposition 2.39 has remained unimproved for several decades, however Villard has recently
 polynomials P and
announced [143] that given two generic bivariate  Q in K[x, y], the bivariate
(2−1/ω) 1+o(1)
resultant Resx (P, Q) can be computed in O (dx dy ) field operations. Since the
computations of resultants are the bottleneck of the algorithms presented in Chapters 3 and 6,
this new algorithm may have a direct impact on their complexity bounds. We discuss this in
the dedicated chapters and sum up the impact of these new bounds in the conclusion.

2.4 Geometric resolution

2.4.1 Bézout bound and multihomogeneity
We expect the complexity of solving polynomial systems to depend on n the number of variables,
m the number of equations, their respective degrees and possibly also their number of solutions.
The number of solutions greatly depends on the system itself and cannot be predicted. However,
it is possible to bound it by the previous data, which is the point of the following definition.
This bound was introduced by Bézout to study the number of points of intersection between
two curves, but it can be generalized as follows:
2.4. Geometric resolution 31

Theorem 2.41 (Bézout’s theorem). Let f1 , . . . , fn be n homogeneous polynomials in n + 1

variables of respective degrees d1 , . . . , dn . Then either the number of projective solutions counted
with multiplicities (in the algebraic closure) is infinite or equal to the product d1 · · · dn .
We will see later that this bound plays a role in complexity results, but we anticipate that
it will not satisfy our needs and we therefore introduce a sharper bound for more structured
systems.
Definition 2.42 (Multihomogeneous system). A multihomogeneous polynomial f is a polyno-
mial such that there exists a partition of the variables in subsets on which the polynomial is
homogeneous. If di is the degree of the homogeneous polynomial with respect to the i-th subset
of variables, the sequence (di ) is called the multi-degree of f .
For example, on K[x1 , . . . , xnx , y1 , . . . , yny ] a bihomogeneous polynomial f of bidegree d1 , d2
is such that

∀λ, µ ∈ K, f (λx1 , . . . , λxnx , µy1 , . . . , µyny ) = λd1 µd2 f (x1 , . . . , xnx , y1 , . . . , yny ).

Definition 2.43 (Multi-degree). We extend this notion to non-homogeneous polynomials by

defining the multi-degree of a polynomial f with respect to a partition of the variables as the
tuple (di ) where di is the degree of f in the variables of the i-th block, when all the other
variables are evaluated at a generic value.
Definition 2.44. [110] Let F = {f1 , . . . , fm } be a non-homogeneous system on K[X1 , . . . , Xn ],
where each Xi = (xi,1 , . . . , xi,ni ) is a tuple of variables and let dj,1 , . . . , dj,n be the multi-degree
of fj with respect to the Xi ’s. The multihomogeneous Bézout number of F is defined as the
coefficient of ni=1 Tini in the product
Q

m X
Y n
dj,i Ti .
j=1 i=1

Theorem 2.45. [110] Consider F as above, then it has no more isolated solutions than its
multihomogeneous Bézout number d.
This bound is much more convenient than the original Bézout bound when dealing with a
system which has a large number of variables appearing with small degree and a small number of
variables appearing with large degree. In Chapter 5 we encounter systems of O(g 2 ) variables with
2
only g variables of “large” degree δ. In this context, the Bézout bound is in δ O(g ) versus δ O(g)
for its multihomogeneous counterpart. The reason why these bounds appear in the complexity
is detailed later on: we will see in Section 2.4 that the cost of computing a geometric resolution
is polynomial in the maximum of the degrees of intermediate ideals, as defined below. However,
contrary to the Bézout bound which is readily computable from the input system, computing
the degree of an ideal is not straightforward. When the input system is generic enough (i.e.
when it is a regular sequence as in Definition 2.26), the degrees of the intermediate ideals can
be bounded by the (multihomogeneous) Bézout bound.
n+1
Definition 2.46 (Degree of an ideal). By identifying a point (λ0 , . . . , λn ) ∈ K with the
polynomial λ0 + λ1 X1 + · · · + λn Xn ∈ K[X1 , . . . , Xn ], there is a dense Zariski open subset
n+1 dim V (I)
O ⊂ (K ) such that for any (`1 , . . . , `dim V (I) ) ∈ O, the algebra K[X1 , . . . , Xn ]/(I +
h`1 , . . . , `dim V (I) i) is a finite dimensional K-vector space of constant dimension, which is called
the degree of I.
32 Chapter 2. Polynomial systems

Definition 2.47 (Reduced sequence). The sequence (f1 , . . . , fi ) is reduced if every intermediate
ideal hf1 , . . . , fj i with j ∈ [1, i] is radical.

Proposition 2.48. Let f1 , . . . , fm be a regular sequence in Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ] and

dx , dy ∈ Z≥0 be such that for any i ∈ [1, m], degx (fi ) ≤ dx and degy (fi ) ≤ dy . Then the degree
of the ideal hf1 , . . . , fm i is at most
!
X m j1 j2
d d . (2.1)
j1 +j2 =m
j1 x y
0≤j1 ≤nx
0≤j2 ≤ny

n
Moreover, this degree is bounded above by 2nx +ny dnx x dy y .

Proof. This is a direct consequence of [123, Prop. I.1] using, with the notation of [123, Prop.
I.1], k = 1, e = 0, P = m, Di,0 = dx , Di,1 = dy , n = nx , n1 = ny . Note that [123, Prop. I.1]
is stated when the base field is C, but the proof works without any major modification when
the base field is a finite field. The last sentence of the statement follows from the fact that the
regularity assumption implies that m ≤ nx + ny , and hence the sum of the binomial coefficients
is bounded above by 2m ≤ 2nx +ny .

2.4.2 Geometric resolutions

Contrary to the previous two sections, we no longer work with multivariate polynomials in dense
representation but as programs describing which operations to perform to evaluate them. We
call that a straight line program (SLP) and give a more precise definition of particular instances
of SLP.

Definition 2.49 (Division-free SLP). A division-free SLP (DFSLP) defined over a field K is a
sequence of polynomials h1 , h2 , . . . , h` ∈ K[X1 , . . . , Xn ] such that each polynomial hi is either a
variable Xt with t ∈ [1, n], an element in K, or hi = hj ◦ hj 0 , where j, j 0 < i and ◦ ∈ {+, −, ×}
is an arithmetic operation. The time of a DFSLP is the total number of arithmetic operations,
and its space is the minimal number of arithmetic registers required to evaluate it. A polynomial
system f1 , . . . , fm is said to be represented by a DFSLP h1 , . . . , h` if {f1 , . . . , fm } ⊂ {h1 , . . . , h` }.

The following lemma gives a bound on the size of a DFSLP needed to represent a bihomo-
geneous polynomial:

Lemma 2.50. Let dx , dy ∈ Z>0 be two positive integers. A polynomial system f1 , . . . , fm ∈

Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ] such that for all i ∈ [1, m], degx (fi ) ≤ dx and degy (fi ) ≤ dy can be
 
nx +dx
ny +dy

represented by a DFSLP with time and space O (dx + dy + m) nx ny .

Proof. There are nxn+d ny +dy

x



x ny monomials µ in Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ] such that degx (µ)
≤ dx and degy (µ) ≤ dy . We consider the DFSLP which starts by evaluating these monomials.
This costs less than nxn+d

ny +dy

x
x
ny (dx + dy − 1) multiplications, using a naive algorithm. Then
we multiply each of these monomials by the corresponding coefficients, and we sum. This costs
m nxn+d

ny +dy
nx +dx
ny +dy

x
x
ny multiplications and m( nx ny − 1) additions.
n
For describing 0-dimensional (i.e. finite) sets V ⊂ Fq where V is defined over Fq , we use a
data structure called a geometric resolution of V . The terminology here is borrowed from [25],
2.4. Geometric resolution 33

see also [64]. The following definition is slightly simpler than the one in [25, Sec. 2.1] because we
restrict ourselves to the 0-dimensional case in the whole thesis (in [25, Sec. 2.1], the definition
is also valid for equidimensional varieties with positive dimension).

Definition 2.51 (Geometric resolution). We say that an Fqe -geometric resolution of V is a

tuple ((`1 , . . . , `n ), Q, (Q1 , . . . , Qn )) where:

• The vector (`1 , . . . , `n ) ∈ Fnqe is such that the linear form

n
`: Fq → F
Pn q
(x1 , . . . , xn ) 7→ i=1 `i xi

takes distinct values at all points in V . The linear form ` is called the primitive element
of the geometric resolution;

• The polynomial Q ∈ Fqe [T ] equals − `(x));

Q
x∈V (T

• The polynomials Q1 , . . . , Qn ∈ Fqe [T ] parametrize V by the roots of the polynomial Q, i.e.

V = {(Q1 (t), . . . , Qn (t)) | t ∈ Fq , Q(t) = 0}.

2.4.3 Computing geometric resolutions

Following [64], we present the main aspects of the computation of a geometric resolution. Let
us consider {f1 , . . . , fn } a system of homogeneous polynomials in K[X0 , . . . , Xn ] such that the
fi ’s form a reduced regular sequence, along with an inequation g 6= 0. The general idea is
to take the equations into account one by one, deducing a geometric resolution of the ideal
Ii+1 = hf1 , . . . , fi+1 i from a geometric resolution of Ii = hf1 , . . . , fi i.
Let us assume that we already have a geometric resolution for Ii , that is a description of the
system Si = {x1 , . . . , xn−i , f1 , . . . , fi } in the form


 xn−i+1 = T

xn−i+2 = vn−i+2 (T ),


q(T ) = 0, ..



 .

xn = vn (T )


The lifting step consists of computing a description of the system

x1 = · · · = xn−i−1 = f1 = · · · = fi = 0, g 6= 0,

in the form 
xn−i+1 = T


xn−i+2 = Wn−i+2 (xn−i , T ),


Q(xn−i , T ) = 0, ..



 .

xn = Wn (xn−i , T )


This step can be seen as seeing the variable xn−i as a parameter of the geometric resolution,
and a solution of Si can be seen as an approximated solution of the above system at precision
O(xn−i ). By the Newton method, this solution can be lifted at precision O(x2n−i ), and the
34 Chapter 2. Polynomial systems

process can be repeated until the precision is sufficient to have an exact resolution. This is
achieved when the precision becomes greater than the degree of the variety.
At the end of the lifting step, the equation fi+1 = 0 is still not taken into account. This is
the point of the so-called intersection step and it is achieved as follows. First, introduce a new
variable X and perform the change of variable in K [[t]]

xn−i = X − txn−i+1 + O(t2 ),

in the previous system. This yields



 xn−i+1 = T

xn−i+2 = Vt,n−i+2 (X, T ),


Qt (X, T ) = 0, ..



 .

xn = Vt,n (X, T )


With Qt a polynomial in X and T and the Vt,j ’s are polynomials in T and rational fractions
in X with coefficients in K [[t]] at precision O(t2 ). Let us now compute

A(X) = ResT (Qt (X, T ), fi+1 (0, . . . , 0, X − tT, T, Vt,n−i+2 (X, T ), . . . , Vt,n (X, T ))).

This resultant is in K[X] [[t]] and substituting X = xn−i + txn−i+1 in A(X) = a0 (X) + ta1 (X) +
O(t2 ), we get
a0 (xn−1 ) = 0, a00 (xn−i )xn−i+1 + a1 (xn−i ) = 0.

Therefore, we have the following geometric resolution for Si ∪ {fi+1 = 0}:



 xn−i = T,



 a1 (T )
xn−i+1 = − 0 = Vn−i+1 (T ),




 a0 (T )
a0 (T ) = 0, ..



 .

a1 (T )

  


 xn = Wn − ,T = Vn (T ).
a00 (T )


This is not completely satisfying as we must still remove the potential solutions contained
in the hypersurface g 6= 0. This is the cleaning step and consists essentially of replacing a0 (T )
by a0 (T )/c(T ), where

c(T ) = GCDT (a0 , g(0, . . . , 0, T, vn−i+1 , . . . , vn )).

2.4.4 Complexity bounds

Theorem 2.52. [64, Th. 1] Let K be a field of characteristic 0 and let f1 , . . . , fn , g be poly-
nomials in K[x1 , . . . , xn ] of degree bounded by d and given in SLP representation of size at
most L. Assume furthermore that the fi ’s define a reduced regular sequence in the open sub-
set {g 6= 0}. The geometric
resolution of the variety V (hf1 , . . . , fn i) \ V (hgi) can be computed
with O n(nL + nω )M(dδ)2 field operations, where ω < 2.38 is the exponent of linear algebra,
δ = maxi=1,...,n deg(hf1 , . . . , fi i) and M(N ) = O(N log2 N log log N ).
2.4. Geometric resolution 35

For our purposes, the main result to remember is that one can compute a geometric resolution
in time polynomial (actually quadratic) in the (multi-homogeneous) Bézout bound. Note that
this result does not apply to our setting, but the following theorem gives a similar statement for
finite fields of sufficiently large size.

Theorem 2.53. [25, Thm. 4.8] Let f1 , . . . , fn ∈ Fqe [x1 , . . . , xn ] be a reduced regular sequence,
where the polynomials are represented by a DFSLP with space S 0 and time T 0 . Set the following
notation:

• The integer d is maxi∈[1,n] (deg fi );

• For any real number x ≥ exp(1), U(x) = x(log x)2 log log x;

• Let δ ∈ Z≥0 be an integer larger than the degrees of the ideals hf1 i, hf1 , f2 i, . . . , hf1 , . . . , fn i.

Assume further that q e ≥ 60 n4 dδ 4 . There is a probabilistic Turing machine using space O((S 0 +
n + d)δ 2 log(q e δ)) and time O((nT 0 + n5 )U(δ)(U(dδ) + log(q e δ))U(log(q e δ))) which takes such
polynomial systems as input and which outputs an Fqe -geometric resolution of the algebraic set
n (M ) (M )
{x ∈ Fqe | f1 (x) = · · · = fn (x) = 0} with probability at least 11/12.

The above complexity estimates derive from two costly steps: the lifting and the intersection.
The former’s complexity is essentially due to the cost of computing a Newton lift at precision
δ and the latter’s bottleneck is the computation of the resultant A. The regularity assumption
on the input system ensures that we have an invertible Jacobian matrix to perform the Newton
iterations. We do not investigate further and refer the interested reader to [64, 25] for more
details.
36 Chapter 2. Polynomial systems
 Chapter 3

Counting points on genus-2 curves

In this chapter, we investigate genus-2 extensions of Schoof’s algorithm both in theory and
practice, along with their applications in cryptography. Like elliptic curves, Jacobians of hy-
perelliptic curves are ideal candidates for cryptographic groups. However, some attacks were
designed for Jacobians of curves of genus ≥ 3 and while these attacks remain exponential, they
imply a less advantageous ratio between key-length and security level. For genus-2 curves, this
ratio is comparable to elliptic curves and by using the Kummer surface associated to the curve
rather than its Jacobian, a genus-2 Diffie-Hellman protocol detailed in [119] can be made faster
than its elliptic analogue [105, 17] thanks to more efficient arithmetic operations designed in [55].
More recently, a signature scheme based on Kummer surfaces of genus-2 curves was designed
in [120]. Almost all the results presented here were already known before the beginning of this
thesis, so this section can be considered as a warm-up for Chapters 5 and 6 as we focus on parts
of the algorithm that we extend later to hyperelliptic curves of larger genera.
Although Pila’s algorithm [115] already yields a polynomial-time algorithm for counting
points on genus-2 curves, the first practical attempt was made in 2000 [57] by combining three
different methods. First, using the Cartier-Manin operator, χ mod p can be computed provided
that the characteristic p is not too large. Since this relates to p-adic methods and will not
be used in this thesis, we do not explore this approach and focus on the two other points:
the computation of χ mod ` for small primes ` in the spirit of Schoof’s algorithm, and the
reconstruction of χ exploiting previous modular knowledge by a baby-step giant-step (BSGS)
algorithm. By then, it was already possible to count points on a curve over a 63-bit prime
field (i.e. in a 126-bit Jacobian) in about two CPU-months. However, generating a Jacobian
of cryptographic size requires much heavier computations that were made possible in [62] by
introducing numerous practical optimizations.
In Section 3.1, we first give an overview of these algorithms along with their complexity
estimates. Section 3.2 reviews practical improvements taken mostly from [62] and how they
were used to compute a cryptographic Jacobian of size 256 bits, i.e. with a 128-bit security
level. A recent note by the NSA [113] advised to upgrade the security level of curve-based
protocols to 192 bits, casting doubt about possibly more efficient yet still exponential attacks
on ECDLP. While finding elliptic curves with this security level is not a problem, it seems quite
a challenge in genus 2. With this motivation in mind, Section 3.1.2 focuses on genus-2 curves
with real multiplication (RM) and how this property is used in [59] to speed-up point-counting.
For non-RM curves, Section 3.3 surveys prospective improvements and ongoing research that
could make it possible to design genus-2 curves that offer a 192-bit security level.

37
38 Chapter 3. Counting points on genus-2 curves

3.1 Genus-2 extensions of Schoof’s algorithm

In this section, C is a genus-2 hyperelliptic curve over a finite field Fq of characteristic p > 2
given by an equation y 2 = f (x), with f monic squarefree of degree 5. The Jacobian of C is
denoted by JC or simply by J when there is no ambiguity. The Frobenius endomorphism is
denoted by π and the characteristic polynomial of its action by either χ or χπ when there is
need for disambiguation.

3.1.1 The Gaudry-Harley-Schost algorithms

We now briefly instantiate properties of hyperelliptic curves from Chapter 1 in genus 2. First,
recall that the characteristic polynomial of π has the form χ(t) = t4 − s1 t3 + s2 t2 − s1 qt + q 2 .
Hence, by the Weil bounds, we are looking for the integers s1 and s2 which respectively satisfy
√
|s1 | ≤ 4 q and |s2 | ≤ 6q. To recover (s1 , s2 ), we compute them modulo ` for sufficiently
many primes `. Given a fixed `, we compute an `-torsion divisor D ∈ J[`] and test whether
−s1 (π 3 (D) + qπ(D)) and π 4 (D) + s2 π 2 (D) + (q 2 mod `)D coincide. If there is only one couple
(s1 , s2 ) satisfying this condition, then we can deduce χ mod ` and move to the next `. Were it
not the case, we apply the same procedure to another torsion divisor D0 to further reduce the
number of candidates for (s1 , s2 ) until only one remains.
Let us now switch to the problem of computing a torsion element. In genus 2, an element
of J is either the neutral element P∞ , the image P − P∞ of a point on C, twice the image of a
point 2(P − P∞ ) or, in most cases, a divisor D = P1 + P2 − 2P∞ with P1 6= ±P2 . In the latter
case, we call such a divisor a “generic” divisor. In general, C does not have a rational torsion
point so that we only look for torsion elements of the last form. In fact, even if there were non
generic divisors other than the neutral element in J[`], Kampkötter showed in [77] that generic
divisors generate J[`] so that it does no harm to miss potential non-generic elements. In genus
larger than 2, we do not know of any similar result and although we still expect torsion divisors
to be generic, we will have to consider some degenerate cases.
Let us consider a generic divisor D = P1 +P2 −2P∞ , with Pi = (xi , yi ) and write a polynomial
system enforcing the fact that D is in J[`]. If `D = 0, then `(P1 − P∞ ) = −`(P2 − P∞ ). We
denote by hui , vi i the respective Mumford forms of both terms, then `D = 0 is equivalent to
u1 = u2 and v1 = −v2 . To develop further, we give a description of the ui and vi , which is a
genus-2 version of the division polynomials defined in [28].

Proposition 3.1. Using the above notation and setting Di = Pi − P∞ , there exist univariate
polynomials d0 , d1 , d2 , e0 , e1 , e2 in Fq [x] such that for ` ≥ 3, the Mumford form hui , vi i of `Di is
given by
d1 (xi ) d0 (xi )
ui (X) = X 2 + X+ ,
d2 (xi ) d2 (xi )
yi
vi (X) = (e1 (xi )X + e0 (xi )) .
e2 (xi )

In the particular case of genus-2 curves, it is known that the respective degrees of these
polynomials are 2`2 − 1, 2`2 − 2, 2`2 − 3, 3`2 − 1, 3`2 − 2 and 3`2 − 3.
Rewriting the equality of the ui , we get the following system in the variables x1 , x2 :

E1 (x1 , x2 ) = d1 (x1 )d2 (x2 ) − d1 (x2 )d2 (x1 ) = 0,

(3.1)
E2 (x1 , x2 ) = d0 (x1 )d2 (x2 ) − d0 (x2 )d2 (x1 ) = 0.
3.1. Genus-2 extensions of Schoof’s algorithm 39

This system is put in triangular form by computing R(x1 ) = Resx2 (E1 , E2 ) and replacing
one of the equations by R(x1 ) = 0. Before doing so, one must actually remove a factor (x1 − x2 )
that appears in both E1 and E2 to avoid having R = 0. This factor is due to the fact that if
x1 = x2 , then we have P1 = ±P2 , thus `(P1 − P∞ ) = ±`(P2 − P∞ ) and therefore u1 = u2 . This is
an example of solutions to our system that do not yield useful information on J[`], we call them
parasites and investigate them later on. Apart from that factor which threatened the validity
of our algorithm, other parasites only increase the complexity by a constant factor. In larger
genus, many more degenerate cases can occur so that a thorough analysis of those parasites is
required.
Once the resultant is computed, a torsion point can be reconstructed as follows: find a root
x1 of R, possibly in an extension of Fq . From the other equation in x1 and x2 , deduce a value for
x2 . Then, there are only four possibilities for (y1 , y2 ), pick one of them to deduce two points P1
and P2 and finally test whether any of the combinations P1 ± P2 leads to a torsion divisor. If it
is not the case, then x1 was a root of a parasite factor of R, so we have to consider another root.
The same can be done if x1 leads to a torsion divisor for which there are still several candidates
for (s1 , s2 ) mod `.
Actually, it may happen that even after checking the whole `-torsion we may end up with
more than one candidate for (s1 , s2 ). When this is the case, one must remember that the (s1 , s2 )
correspond to polynomials that annihilates the Frobenius action. Computing their GCD, we
can first deduce a multiple of its minimal polynomial. Luckily, the degree and roots of that
polynomial are enough information to recover the actual characteristic polynomial χ mod `.
Since this is unlikely, we do not detail that subtlety and refer to [62, Sec. 3.4] for that matter.
It is possible to eliminate all the parasites by taking into account all the equations and not
only the first two. Writing that the v-coordinates of `(P1 − P∞ ) and `(P2 − P∞ ) have to be
opposite amounts to the third equation

E3 (x1 , x2 ) = e1 (x1 )e2 (x1 ) − e1 (x2 )e2 (x1 ). (3.2)

Then, one can compute R1 = Resx2 (E1 , E3 ) and apply the previous method to R̃ = gcd(R, R1 )
instead of R.
Following the D5 strategy of [40] mentioned in Section 1.2.2.0, one could actually recover
a triangular form of the `-torsion ideal I` , and perform operations in the quotient ring while
handling the potential “forbidden divisions” by removing the vanishing factor from the univariate
polynomial of the lex Gröbner basis of I` . This approach was considered but not used in [60]
as the first strategy seems more efficient for primes smaller than 19. In [62], the D5 strategy is
preferred as ` goes up to 31.
To do so, we compute both the resultant and subresultant of the equations E1 and E2 to
put the system in the form
S0 (x1 ) + x2 S1 (x1 ) = 0,
R(x1 ) = 0.
Then, taking into account the equation E3 , we clean up the parasites by computing R̃ and the
modular inverse S̃ = S0 /S1 mod R̃. We can therefore represent the `-torsion ideal by the base

y22 −f (x2 )
y12 −f (x1 )
x2 +S(x1 )
R̃(x1 ).
40 Chapter 3. Counting points on genus-2 curves

Once given such a representation, it is no longer necessary to factor R, as one can consider
a generic D` = (x1 , y1 ) + (x2 , y2 ) − 2P∞ in Fq [x1 , x2 , y1 , y2 ] and test the equation χ(D` ) = 0 in
Fq [x1 , x2 , y1 , y2 ]/I` .

input : A genus-2 hyperelliptic curve C given by a monic squarefree f ∈ Fq [x] of

degree 5
output: The characteristic polynomial of the Frobenius
w ← 1;
while w ≤ 12q do
` ← NextPrime(`) ;
w ←w·` ;
Compute R(x1 ) = Resx2 (E1 , E2 ) and S0 (x1 ) + x2 S1 (x1 ) = Subresx2 (E1 , E2 ) ;
R̃ ← GCD(R, Resx2 (E1 , E3 ));
S ← S0 /S1 mod R̃;
Deduce a basis for I` ;
Set D` a generic divisor in Fq [x1 , x2 , y1 , y2 ]/I` ;
Eliminate candidates (s1 , s2 ) ∈ (Z /` Z)2 for which χ(D` ) 6= 0 ;
Deduce (s1 , s2 ) mod `;
end
Perform a CRT to recover the actual (s1 , s2 ) ;
return χ(t) = t4 − s1 t3 + s2 t2 − s1 qt + q 2
Algorithm 4: Genus-2 point counting algorithm from [57, 62, 60]

We now follow the complexity analysis of [57, 62]. Note that [57] originally proves a com-
9
plexity in O(log
e q) because it does not make use of fast arithmetic in Fq .

8
Theorem 3.2. [57, Sec. 5.4] Algorithm 4 has a complexity in O(log
e q) bit operations and a
5
memory requirement in O(log q).

Proof. Compared to the rest of the algorithm, computing the genus-2 division polynomials takes
negligible time and memory in practice: even a naive approach using the recurrence formulas
of [28, Eq. (1.8)] and storing each `-division polynomials yield a complexity in O(`3 log q) memory
bits and O(`3 log q) binary operations (each step requires O(1) operations on polynomials over
Fq with degree in O(`2 )) which is within the complexity bounds we aim for.
Computing the bivariate resultant R is done by an evaluation / interpolation scheme. The
degrees of E1 and E2 in the xi are in O(`2 ) so that by Proposition 2.39 the polynomials R, S1 , S2
can be computed in O(` e 6 ) field operations using O(`4 ) interpolation points, i.e. O(`4 log q) bits
of memory. Since we consider polynomials of degrees in O(`4 ), the GCD computations also fit
e 4 ) field operations.
within O(`
e 4 ) field operations and each ele-
In the algebra Fq [x1 , x2 , y1 , y2 ]/I` , each operation costs O(`
ment is stored on O(`4 log q) memory bits. Finding χ mod ` costs at most O(`) operations in the
algebra Fq [x1 , x2 , y1 , y2 ]/I` and a constant number of Frobenius computations, which amounts
e 4 (log q + `)) field operations. During this step, only a fixed (i.e. independent of `) number
to O(`
of elements needs to be stored, hence a memory requirement in O(`4 log q) bits.
Since both the number of primes ` and the size of the largest ` to consider are in O(log q) and
that each operation in Fq has a bit complexity in O(log e q), we deduce the final bit complexity
8 5
in O(log
e q) and a memory requirement of O(log q) bits.
3.1. Genus-2 extensions of Schoof’s algorithm 41

8
The complexity in O(log
e q) bit operations is much larger than that of Schoof’s algorithm in
5
O(log q) and the exponent is twice larger than that of the SEA algorithm. It is very challenging
e
to get modular information on χ for prime numbers above 30, which is the reason why other
strategies are used in practice to terminate the computations. This complexity analysis also
reveals an interesting phenomenon: compared to Schoof’s algorithm, applying powers of π to
a generic torsion element is no longer the bottleneck in the genus-2 case. Indeed, the most
costly step is the computation of a triangular form for the `-torsion ideal. When g grows, it is
even more conspicuous that this step is also the bottleneck of our generalizations of Schoof’s
algorithm.
Confronted with such a complexity bound, one may look for more favorable instances of
the problem in which the bounds are more reasonable. For example, one would like to find
Jacobians with “a smaller torsion”. Unfortunately, such Jacobians cannot exist as they must
satisfy Proposition 1.35. However, we will see in the next section that there exist families of
curves whose torsion can be split into a direct sum of subspaces which are similar in size to the
`-torsion subgroup of an elliptic curve. Such subspaces correspond to ideals of smaller degrees
than the `-torsion ideal, and therefore putting them in triangular form is less costly than doing
the same to the full `-torsion.

3.1.2 The case of RM curves

In this section, we review how the previous point-counting algorithm can be adapted into a
faster one when applied to families of curves that are equipped with a particular endomorphism.
This is work of Gaudry, Kohel and Smith in [59] and it will be extended to genus-3 curves in
Chapter 6.
Since we consider hyperelliptic curves defined over Fq , the Frobenius yields an endomor-
phism of their Jacobians, so that Z[π] ⊂ End(J). Furthermore, the dual π ∨ of the Frobenius
endomorphism is also in End(J) hence any curve has RM by F = Q(π + π ∨ ) in the sense of Def-
inition 1.40. However, this RM is not efficient because applying the endomorphism ψ = π + π ∨
to a point of C costs O(log q) operations in J as it involves q-th powers. Worse, this RM is not
explicit in the sense that we do not have formulas to describe the action of ψ on the curve. For
our purpose, we ask for curves with an additional endomorphism that is easy to compute in the
sense of the following definition.

Definition 3.3. Let η be a real element of a number field, and let C be a hyperelliptic curve
with RM by Z[η]. We say that the real multiplication is explicit if we have explicit formulas to
compute the Mumford form η(P − P∞ ) for P = (x, y) the generic point on the curve C.

Remark Consider Q(π) the so-called CM-field of J, then the intersection Q(π) ∩ EndFq (A) is
an order O of Q(π) and hence it is a subring of the maximal order OQ(π) . By a result from [145],
O also contains a “minimal order” as it has to contain Z[π, π ∨ ].
Let us consider a genus-2 curve C with explicit RM by Z[η] as in Definition 3.3. Examples
of such curves are given by the family Ct : Y 2 = X 5 − 5X 3 + 5X + t from [138] with RM by
Z[ζ5 + ζ5−1 ], as well as other families due to Humbert and Mestre [101]. They are detailed in [87]
along with examples of RM in higher genus.
In what follows, we assume that the curve C has explicit RM by Z[η]. Let us denote ψ = π+π ∨
and recall the expression of χπ (t) = t4 − s1 t3 + s2 t2 − s1 qt + q 2 , from which we deduce the
characteristic polynomial of ψ,
χψ (t) = t2 − s1 t + s2 .
42 Chapter 3. Counting points on genus-2 curves

By the previous remark, Z[ψ] ⊂ Z[η] hence there exist two integers a and b such that
ψ = a + bη. They are uniquely determined by s1 and s2 because

s1 = Tr(ψ) = 2a + b Tr(η), and s2 = N (ψ) = a2 + ab Tr(η) + b2 N (η). (3.3)

Contrary to Section 3.1 we do not test directly the characteristic equation of π but the
equality between the last two members of ψπ = π 2 + q = aπ + bηπ. In other terms, we compute
a mod ` and b mod ` by finding ā and b̄ in (Z /` Z) such that for any torsion divisor D we have

āπ(D) + b̄ηπ(D) = π 2 (D) + (q mod `)D.

This brings two advantages over the general case: first, we only have to apply powers of π up
√
to π 2 instead of π 4 , and more importantly, [59, Eq. 10] shows that both a and b are in O( q)
√ √
while s2 is in O(q). More precisely, one can prove that |a| ≤ 4 q and |b| ≤ 2(| Tr(η)| + 1) q.
Yet, this improvement only reduces the number (and size) of primes ` to consider by a
constant factor since it depends logarithmically on the width of the Hasse-Weil interval. The
most significant gain lies in the structure of the `-torsion, which allows us to find torsion divisors
D more easily.
Let us consider a prime ` that splits in Z[η] into the product p1 p2 . We first detail how this
can be used to split J[`] ' (Z /` Z)4 into a direct sum of two subspaces isomorphic to (Z /` Z)2 .
In [59], the ideals pi are assumed to be principal because the order Z[η] has class number 1
in all the examples of RM families. This assumption is not necessary and will be removed in
Chapter 6 although it still holds in the genus-3 RM family that we used for practical experiments.
For simplicity, we follow [59] and make the same assumption in this chapter.

Lemma 3.4. [59, Lemma 1] If p is a principal ideal of norm ` in a real quadratic order Z[η], √
then there exists an effectively computable generator α = a + bη of p with both a and b in O( `).

Computing small generators α1 and α2 of p1 and p2 , we have J[`] = J[α1 ] ⊕ J[α2 ] so that
any torsion divisor D can be written D1 + D2 with Di ∈ J[αi ]. We have therefore transformed
the problem of finding a generic element of `-torsion into finding a generic element of αi -torsion.
To do so, we proceed exactly as in Subsection 3.1.1 but with the equation αi (Di ) = 0 instead
of `Di = 0. Once found Di a generic element of J[αi ], we compute π 2 (Di ) + (q mod `)Di and
kπ(Di ) for any k ∈ Z /` Z to find ki the only value of k such that these two quantities are equal.
This is summed up in Algorithm 5.
5
Theorem 3.5. [59, Th. 1] Algorithm 5 has a complexity in O(log
e q) bit operations.

Proof. Let us consider a fixed prime ` that splits in Z[η]. For each pi , we √ compute a small
generator αi = βi + γi η as in Lemma 3.4. Since we know there is one of size O( `), it is possible
to find it by exhaustive search for O(`) field operations.
We now compute a generic element Di ∈ J[αi ] using the strategy of Section 3.1.1 except that
we write √ αi (P1 − P∞ ) = −αi (P2 − P∞ ) instead of `(P1 − P∞ ) = −`(P2 − P∞ ). Since βi and γi
are in O( `), the Mumford form of αi (Pi ) have coefficients whose degrees in the abscissa xi is in
O(`). Then, following the analysis in the proof of Theorem 3.2 with equations of degree ` instead
of `2 , we prove that a generic element of Ker αi can be computed in O(` e 3 ) field operations.
Then, finding the ki ’s require two applications of π and hence O(log q) field operations and
e
at most O(`) field operations for the exhaustive search. Deducing (a, b) from k1 and k2 is linear
algebra in F` , which is negligible, and therefore each step in the main loop of Algorithm 5 has
e 3 ) field operations i.e. O(`
an overall cost of O(` e 3 log q) bit operations.
3.2. Practical improvements and past results 43

input : q an odd prime power, and f ∈ Fq [X] a monic squarefree polynomial of degree
5 such that the curve Y 2 = f (X) has explicit RM by Z[η].
output: The characteristic polynomial χπ ∈ Z[T ] of the Frobenius endomorphism on
the Jacobian J of the curve.
w ← 1;
` ← 2;
√
while w < max(|2 Tr(η)| + 1, 4) q do
Pick the next prime ` that splits in Z[η] ;
Compute the ideal decomposition ` Z[η] = p1 p2 , corresponding to the eigenvalues
λ1 , λ2 of η in J[`] ;
for i ← 1 to 2 do √
Compute a small generator αi of pi with coefficients in O( `) ;
Compute a generic element Di in J[αi ] ;
Find the unique ki ∈ Z /` Z such that ki π(Di ) = π 2 (Di ) + qDi ;
end
Find the unique (a, b) in (Z /` Z)2 such that a + bλi = ki , for i in {1, 2} ;
w ← w · `;
end
Reconstruct (a, b) using the Chinese Remainder Theorem ;
Deduce χπ from Equations (3.3).
Algorithm 5: Overview of the genus-2 RM point-counting algorithm from [59]

Since the RM field is fixed, by Chebotarev’s density theorem, half of the primes split in Z[η]
and thus both the number of primes ` to consider and the size of the largest one are still in
O(log q). Replacing ` by log q and adding a factor log q for the number of primes proves the
theorem.

In this section, we have actually given a simplified version of point-counting algorithms to

simplify the exposition and the complexity analysis. In practice, more optimizations have to
be done to achieve significant practical results, in particular when using Algorithm 4. These
modifications are the focus of the following section.

3.2 Practical improvements and past results

In this section, we review practical improvements to Algorithm 4. None of them change the
8
complexity in O(log
e q) bit operations but some enabled to halve the number of operations.
We conclude by reviewing the efforts made to design a genus-2 curve with 128-bit security
level, which took great advantage of the practical improvements. Both the improvements and
computations were made prior to this thesis, in the early 2010’s so we also give a panorama of
what has changed since then in the next section.

3.2.1 Sharper modelling

Eliminating “parasites”
In Section 3.1.1, we already mentioned the possibility of parasites, i.e. extraneous factors of
the resultant coming not from torsion points but from particularities of the input system. More
44 Chapter 3. Counting points on genus-2 curves

2 −2
than the factor (x1 − x2 ), it is pointed out in [57] that d2 (x1 )2` divides the resultant R(x1 ).
Indeed, recall the equations

E1 (x1 , x2 ) = d1 (x1 )d2 (x2 ) − d1 (x2 )d2 (x1 ) = 0,

(3.4)
E2 (x1 , x2 ) = d0 (x1 )d2 (x2 ) − d0 (x2 )d2 (x1 ) = 0,

and remark that if d2 (x1 ) vanishes, then any root of d2 is also a common root of E1 (x1 , ·) and
E2 (x1 , ·).
Using evaluation / interpolation techniques, we can directly avoid these parasites by evaluat-
2
ing R̃(a1 ) = Resx2 (E1 (a1 , x2 ), E2 (a1 , x2 ))/d2 (a1 )2` −2 and then reconstruct R̃ by interpolation.
Knowing the degrees of the di , we see that deg R̃ = 4`4 − 10`2 + 6 is about twice smaller than
deg R. The computation of R representing most of the time spent by the algorithm, this trick
almost halves the running time.
In [62], it was further noticed that d2 has the form f 3 δ 2 , with f the defining polynomial of
the hyperelliptic curve. This also results in parasites that can be eliminated, albeit with a less
spectacular decrease of deg(R̃) by roughly `4 /4.

Resymmetrization
Since the `-torsion has size `4 , we know that R̃ still has about three times more parasitic factor
than factors coming from actual torsion points. Notice that System 3.1 is symmetric in the
variables x1 and x2 . In [60] the symmetry is used and the change of variables U1 = −(x1 + x2 )
and U0 = x1 x2 allows to halve the degrees of the input system. Moreover, the parasites of the
previous paragraph can still be tracked after resymmetrization so that an additional factor 2 is
gained in the degree of R̃.

3.2.2 Further optimization

Lifting the `-torsion
The cost of computing χ mod ` for an additional ` being in O(` e 6 log q), one may wish that there
were “more small primes” to retrieve modular information. The idea behind Algorithm 4 still
works when replacing ` by `k . Indeed, for k > 1 we may try to find a generic `k -torsion divisor
Dk and test whether χ(Dk ) = 0. The modelling is done iteratively by solving systems of the
form `D = Dk−1 with Dk−1 an `k−1 -torsion divisor found previously and D a divisor whose
coordinates are the indeterminates. This was introduced in [57] for ` = 2 and then extended
for primes up to 7 in [60, 62]. In [62], the largest ` was constrained by memory and powers of
small ` were used until the computations became too heavy. Today, the memory is no longer
a problem but running practical tests seems to be the only way of making one’s mind on the
subject as it is hard to assess the difference between going for `k or trying an additional prime
`0 of similar size. Since we do not use the torsion lifting technique thoroughly in the remainder
of this thesis, we do not detail this further and refer to [62, Sec. 4] for more information.

Eliminating unpredictable parasites

Even after resymmetrization and removal of predictable parasites, deg R̃ is still almost twice
larger than what it should be. To eliminate the remaining parasites, it is possible to compute
a third equation E3 involving the v-coordinates of the respective Mumford forms of `(Pi − P∞ )
as in [57], compute its resultant R1 with either E1 or E2 and then the GCD of R and R1 ,
which is of the right degree (`4 − 1) in practice. Back then, it was unclear whether this step
3.2. Practical improvements and past results 45

was preferable to searching directly factors in R̃ with risks of useless computations leading to
parasite solutions, or “cleaning” R̃ before factorization to ensure that no parasite remains, at
the cost of almost doubling the running time. This was ruled out in [62, Sec. 3.4] by using an
alternative strategy involving the recovery of a Gröbner basis for the whole symmetrized torsion
ideal, and by designing another way of cleaning the parasites.
Let S = S0 /S1 mod R̃, and let us consider the algebra B = Fq [U1 , X]/hR̃(U1 ), X 2 + U1 X +
S(U1 )i. Let P1 and P2 be points of respective abscissae X1 = X and X2 = −U1 −X, and compute
`(P1 −P∞ ) and `(P2 −P∞ ) in the algebra B. Note that we can handle these scalar multiplications
without having to worry about the ordinates and only dealing with their respective squares
f (X1 ) and f (X2 ). Ultimately, we want `(P1 − P∞ ) = −`(P2 − P∞ ) so that the v coordinates
of their Mumford forms have to be opposite. Denote v1i X + v0i such Mumford forms. We
must have v112 = v 2 for D = P + P − 2P
12 ` 1 2 ∞ to be an `-torsion element. Experimentally,
2 − v 2 ) has no remaining parasite factor.
R = gcd(R̃, v11 12
Let us now explain how to recover χ as in the non-resymmetrized case. Let us consider
D` = hX 2 + U1 X + U0 , V1 X + V0 i, a generic divisor in Fq [U1 , U0 , V1 , V0 ]. The first equations
that D` must satisfy to be an `-torsion divisor are R̃(U1 ) = 0 and U0 = S(U1 ), which determine
its u-coordinate. Now, remark that writing D` = P1 + P2 as above, the coordinate V1 must be
(Y1 − Y2 )/(X1 − X2 ) and the quotient V0 /V1 must be (X1 Y1 Y2 − X2 f (X1 ))/(Y1 Y2 − f (X2 )). We
can actually find expressions involving only the X-coordinates for Y1 Y2 and for Yi2 = f (Xi ). For
the V1 -coordinate, however, we have to consider its square which we express as

f (X1 ) + f (X2 ) − 2Y1 Y2

.
(X1 − X2 )2

Plugging back the expressions of the Xi ’s in terms of the Ui ’s, we end up with expressions
of the coordinates of D` involving only U1 and U2 , which yield the following Gröbner basis for
the torsion ideal I` ⊂ Fq [X1 , X2 , V1 , V0 ]:

V0 −V1 Z(U1 )
V12 −W (U1 )
U0 −S(U1 )
R̃(U1 )

with all the polynomials R̃, S, W, Z of degrees ≤ (`4 − 1)/2. We refer to [62, Sec. 3] for more
information about this process.
Once given such a representation, we likewise avoid the factorization of R̃, and directly test
the equation χ(D` ) = 0 in Fq [U1 , U0 , V1 , V0 ]/I` , using the D5 strategy to avoid division by any
non-invertible element in that algebra.

3.2.3 Final collision search

In Section 1.2.2, we mentioned the fact that it is possible to directly test the characteristic
equation of the Frobenius in the whole Jacobian to compute (s1 , s2 ). Although the complexity
of such an approach is exponential, it is still very efficient in practice. Moreover, it can benefit
from the knowledge of (s1 , s2 ) modulo an integer m and it can also be run in parallel. A one-
dimensional approach when looking only for #J is to take random divisors D ∈ J and compute
their orders to deduce factors of #J until only one possibility remains in the Hasse-Weil interval.
Since Jacobians of hyperelliptic curves are often “almost cyclic”, a random D has a large order
46 Chapter 3. Counting points on genus-2 curves

and only a few random D are expected to be necessary to determine #J. This method a priori
gives the exponent of the group rather than its order, but it can be adjusted to deduce the
actual order of the group, as presented in [33][Algorithm 5.4.1]. Note that this requires bounds
on the cardinal of the input group, which is not a problem in our case since they are provided
by the Weil bounds.
Using a birthday paradox approach, one expect to find (s1 , s2 ) with running time and memory
requirement in O(q 3/4 ). If s1 and s2 are already known modulo an integer m, then the search-
space is reduced by a factor m2 and the complexity by a factor m. An idea introduced in [99] is
to split the characteristic equation into two parts: one depending only on a parameter derived
from s2 and the other depending on two parameters derived from s1 and s2 . Then, instead of
directly trying random values for these 3 parameters, one stores all the possible values for the
first part, and deduces bounds for the two other before performing a random collision search to
determine the remaining two parameters. The main drawback of these methods is the storage
requirement.
The key to avoiding storage is to look for collision of deterministic sequences which are
assumed to behave as pseudo-random sequences in the complexity analysis (an assumption that
is backed by practical evidence). This is inspired by Pollard’s kangaroos method. To simplify
√
the exposition, let us first assume that m ≥ 8 q, so that s1 is already completely determined
and we only look for the right value of s2 . Let us split s2 = s¯2 + ms˜2 with s¯2 already known.
Denote by K = q 2 + 1 − s1 (q + 1) + s¯2 so that #J = K + ms˜2 . From the bounds on s2 we deduce
bounds on s˜2 and we actually substract from K some multiple of m and s˜2 by some constant
to make these bounds of the form |s˜2 | ≤ B. Let us pick a random D ∈ J and define the “wild”
and “tamed kangaroos” as

W = {(K + mσ2 )D | |σ2 | ≤ B} and T = {mσ2 D | |σ2 | ≤ B}.

When an element of W ∩ T is found, we are able to compute s˜2 . Using a birthday paradox
approach, we want to compute random elements in each set until we find an element in the
intersection. But we need a way of storing some elements in order to detect such a collision.
Storing any of the two sets in totality is excluded since it would entail memory requirements
comparable to the algorithm of [99]. A workaround is to use distinguished points, i.e. elements of
J with a particular feature, such has having the 20 last bits of their u0 coordinate equal to zero.
We now fix a hash function on divisors and perform pseudo-random walks (Di ) such that Di+1
is determined by the hash of Di and decide to stop the pseudo-random walk whenever it hits a
distinguished divisor. We perform many such walks in W and T and only store one element per
walk i.e. the final distinguished divisor. Due to the deterministic design of the pseudo-random
walks, if a walk in W collides with a walk in T , then they keep colliding until the end. Hence,
the distinguished divisor that is stored is also a point in W ∩ T , which is the reason why only
the last element of the walk has to be stored. We redirect to [60] for complexity analysis and
optimization of the parameters. Note that similarly to the one-dimensional approach we a priori
only get s2 modulo the order of D, but once again this can be fixed using [33][Algorithm 5.4.1].
In the general case, however, both s1 and s2 are not completely known and the previous
collision has to be sought in intersecting rectangles instead of intervals. Each step is made in
a plane instead of a line although in practice it is better to impose a fixed proportion of one-
dimensional steps in the direction corresponding to s2 , as it is much larger than s1 . Indeed, our
√
rectangles are really flat because of the Weil bounds in O( q) for s1 and in O(q) for s2 . To
perform the pseudo-random walk, Di+1 is computed from Di by adding an offset of the form
(−1)b α(q + 1)mD + βmD, where b, α and β depend on the hash of Di . These quantities are
3.2. Practical improvements and past results 47

initially taken uniformly at random respectively in {0, 1}, [0, 2L1 ] and [0, 2L2 ] with the Li ’s being
parameters. Note that once each tuple (b, α, β) is associated to a value of the hash function, it
remains the same in order to keep our walks deterministic.
Let us now discuss on the parameters involved. First, let N be the number of points in the
rectangle {(s1 , s2 ) | b1 ≤ s1 ≤ B1 , b2 ≤ s2 ≤ B2 }. Let C be the number of chains to create: this
is fixed by the user and must be large enough to avoid each chain √ being too long but small enough
not to require too much memory. We expect to construct O( N ) points before a collision, and
the user can estimate an actual value λ for this quantity. We must now decide of the probability
pD for a random divisor to be distinguished: too small will imply a larger running time while too
large will be too demanding on memory. We can actually relate it to C: if we are too compute
about λ points divided into C chains, then we expect each chain to have a length about λ/C.
Since a chain ends when it reaches a distinguished point, its expected length is 1/pD . Equating
the two quantities yields pD = C/λ. To fix the parameters L1 and L2 , let us observe that we
do not want the chains to leave the intersection of the rectangles, because there is no hope to
find a collision outside. On average, each chain goes a distance L2 /pD from the center along the
s2 -axis so we want L2 /pD to be small enough compared to B2 − b2 , for example one tenth of it.
We do the same for L1 but warn that along the s1 -axis, the chains can move in both directions
because of the b
p sign (−1)√. In that case, by the central limit theorem we expect the chains to be
at distance 2 2/3πL1 / pD of the center along the s1 -axis. Setting L1 to be about one tenth of
√
the limit and approximating the previous term, one can set L1 = (B1 − b1 ) pD /9. Once again,
we refer to [60] for a heuristic complexity analysis and discussion on the choice of parameters.
We give more details about that in Chapter 6 in the tridimensional case.
Note that the running time of Gaudry and Schost’s algorithm for one- and two-dimensional
collision search depends on the overlap between the sets T and W . In [52], Galbraith and Ruprai
propose a more detailed complexity analysis as well as an improved version of the algorithm in
which the size of W ∩ T is constant.

3.2.4 A cryptographic genus-2 curve

In [62, Sec. 5], a cryptographic genus-2 curve is found after large scale computations involving
a million CPU hours. More precisely, this curve is defined over F2127 −1 and both its Jacobian
and that of its quadratic twist have order 16 times a large prime. Further properties ensuring
efficiency of the group law of the associated Kummer surface are imposed but we do not describe
all the details here. In particular, these conditions entail some rationality conditions that are
responsible for the factor 16 in the cardinality of the Jacobian. To find a “random” curve in
the sense that it has no additional remarkable property which could decrease its security level,
many curves are generated and those among them who do not satisfy the requirements are
discarded until one suitable curve is found. The Kummer surfaces associated to these curves
can be parametrized by four parameters called theta constants. The starting set was chosen to
be the set of curves whose associated theta constants have squares between −40 and 40. Among
them, 83639 lead to Kummer surfaces with nice arithmetic properties.
Then, Schoof’s algorithm is applied to all these curves but with an early-abort strategy
ensuring that we first compute the order of J modulo small primes ` and discard curves such
that #J mod ` = 0. Filtering out with ` = 3 and ` = 5 leaves “only” 21201 candidates.
Computing #J mod 32 and #J mod 7 reduce that number to 3608. Schoof’s algorithm was
continued using this early abort approach until ` = 31, for which #J is known modulo ' 230 .
Back then, memory requirements were too high to go further and powers of primes were used up
to 217 , 37 , 54 and 72 , allowing knowledge of #J modulo N ' 277 . In the end, the actual (s1 , s2 )
48 Chapter 3. Counting points on genus-2 curves

of 586 curves were computed for a total time of roughly 1000 CPU hours per curve, using the
collision search algorithm described in Section 3.2.3. The 128-bit security level Jacobian that
was retained corresponds to a hyperelliptic curve defined over F2127 −1 by the equation

y 2 =x5 + 64408548613810695909971240431892164827x4
+ 76637216448498510246042731975843417626 x3
+ 154735094972565041023366918099598639851 x2
+ 9855732443590990513334918966847277222 x
+ 81689052950067229064357938692912969725.

It has since been used in various cryptographic implementations and records such as [119, 18,
120].
To our knowledge, this example is still the only random 128-bit secure genus-2 curve in the
literature and this is no wonder because of the efforts required to achieve it. Worse, to hope for
a higher security level, one needs to compute modular information for larger `, for a complexity
e 6 log q). The goal of the next section is to survey the prospects for larger cryptographic
in O(`
genus-2 Jacobians. Note that there are other ways of finding such Jacobians by using the CM
method or by restricting to curves with RM, but one could prefer a less structured curve as
additional properties might well lead to faster attacks on the DLP, although none have been
published yet.

3.3 Prospective improvements

3.3.1 Feasability of a cryptographic 384-bit Jacobian
In [62], when looking for a 256-bit Jacobian of a genus-2 curve, the Schoof-like part had to be
halted at ` = 31 and further modular information up to 77 bits was extracted from non-prime
torsion. Counting points on a single curve over a 192-bit field without using additional prime `
would require a collision search over a space of size about 2140 and thus about 270 operations.
As this task has to be repeated over several hundred curves, this would not be reasonable.
On the other hand, counting points on a curve over a 192-bit field without using an expo-
nential step would require to perform Schoof’s algorithm up to ` = 149, or ` = 109 if we manage
to recover the same amount of information as [62] by lifting the `-torsion. Assuming that the
collision search algorithm brings about 250 bits of information on (s1 , s2 ), we can decrease the
largest ` to 79 or 73.
The previous limit was set at ` = 31 because of a lack of memory. Since time complexity
grows in O(` e 6 log q) while memory grows in O(`
e 4 log q) when ` grows, we expect the memory
requirements to be less of a concern. It seems to be the case indeed as our simulations lead to
estimate the memory requirements to be under 500 GB even for an ` as large as 83. The running
time, however, quickly grows beyond control as, for instance, we expect that the computation
of (s1 , s2 ) modulo primes ` up to 73 would require about 10000 CPU days.
Even up to ` = 53, such computations would take about 1000 CPU days, more than 80 times
the whole time needed to compute the previous cryptographic Jacobian. Worse, even assuming
the modular knowledge up to ` = 53 and torsion lifting identical to [62], the collision search
would still have to cope with a search space of size ' 295 .
In practice, almost all the running time would be spent doing either evaluations using resul-
tant computations or collision search. These two steps being parallelizable, such computations
3.3. Prospective improvements 49

may not be completely impossible. However, we question the point of spending conspicuous
amounts of computational power that might even not be negligible compared to the cost of
discrete logarithm computations in the secure curve.
Unless further improvements are made, it seems that the only plausible alternatives for safe
genus-2 curves come from RM curves or from the CM-method. Indeed, in [59] counting points
on a RM-curve defined over a 512-bit prime field is done in about 80 CPU days. When using
the CM-method, the order of the Jacobian is almost already determined and the bulk of the
computations is actually to find suitable fields K and Fp and recover an equation of a curve
C over Fp with CM by the ring of integers of K. When the CM-field K has a small class
number as in [146], Jacobians of genus-2 curves offering a 128-bit security level can be computed
in a matter of minutes. Later on, further examples with fields of larger class numbers were
constructed in [44], the largest one being the field K = Q[X]/(X 4 + 1357X 2 + 3299), with class
number 40032. In the next two subsections, we discuss research areas that could help make
random genus-2 curves competitive again.

3.3.2 Further improvements

For prime numbers ` larger than 30, we observe that computing the bivariate resultant represents
more than 90% of the running time of Algorithm 4, which is no surprise since we expected it to
be the bottleneck asymptotically. Therefore, to improve the running-time of this algorithm, one
must either reduce the size of the input system or find a faster way of computing the resultants.
Efficient computation of resultants is a problem that has drawn a lot of attention in the past
decades and for which there has been no significant improvement in the previous years. Recently,
however, Villard proposed a faster algorithm [143] for computing bivariate resultants that we
already mentioned at the end of Section 2.3. Using this algorithm, the cost of the computation
of bivariate resultants could be decreased to O(`6−2/ω+o(1) ), which represents an improvement
by a factor at least (log q)2/3 in the final complexity. Recall that ω is the exponent from linear
algebra, which was proven to be smaller than 2.38. In practice however, we expect to be using
Strassen’s algorithm for matrix multiplication and thus have a value of approximately 2.8 for ω.
Also note that Villard’s algorithm relies on some genericity assumption which may prevent us
from using it in our case.
In order to reduce the size of the bivariate equations, a possibility could be to forecast and
remove additional parasites. This approach, however, seems to have been fully explored in [62].
Note that since deg R̃ is reduced to about 2`4 in [62] and has to be at least `4 /2 to encode
the whole `-torsion, no more than a factor 4 can be saved anyway. Another way to reduce the
degrees of our equations could be to consider less than the full torsion, as in the SEA algorithm
for which factors of division polynomials of degree O(`) are considered instead. While we still
lack the tools to make this technique a reality in genus 2, the next section reviews ongoing
research in that direction.

3.3.3 Generalization of the Elkies-Atkin improvements

In the elliptic case, Schoof’s algorithm has been improved by Elkies and Atkin, as detailed
in [128]. Both improvements involve the so-called modular polynomial Φ` (X, Y ) which is a
bivariate polynomial defined by the property Φ` (j(E1 ), j(E2 )) = 0 if and only if the curves E1
and E2 are `-isogenous.
Given a curve E, the univariate polynomial Φ` (j(E), X) has a very constrained factorization
pattern in Fq [X]. Indeed, only three possibilities occur for the degrees of the irreducible factor-
50 Chapter 3. Counting points on genus-2 curves

ization Φ` (j(E), X) = f1 · · · fs . We denote by (δ1 , . . . , δs ) the tuple formed by the degrees of the
fi ’s rearranged in non-decreasing order and we use the terminology of [8] to classify the primes
` according to the tuple associated to Φ` (j(E), X):

• if it is (1, `), we say that ` is a volcanic prime

• if it is (1, 1, r, . . . , r), we say that ` is an Elkies prime

• if it is (r, r, . . . , r), we say that ` is an Atkin prime

The improvement by Atkin allows to deduce information on χ mod ` from this factorization
pattern: it does not change the asymptotic complexity of Schoof’s algorithm, but provides a
significant speed-up. Indeed, we have χ(X) = X 2 − tX + q and Atkin proved that t2 mod ` is
either 4q mod ` in the volcanic case or (ζ + ζ −1 + 2)q mod ` in the other two cases, with ζ a
primitive e-root of unity, for e dividing either ` + 1 if ` is an Atkin prime or ` − 1 if ` is an Elkies
prime.
The improvement due to Elkies consists of determining t mod ` by replacing the test χ(P ) = 0
in E[`] by the test π(P ) = λP in the kernel of an `-isogeny determined by the factorization of
Φ` (j(E), X). Since the kernel is given by a polynomial of degree (` + 1)/2 versus (`2 − 1)/2
for the `-division polynomial, this decreases the complexity of computing χ mod ` by a factor
O(`) provided that there exists an `-isogeny. This is the case when ` is either a volcanic or an
Elkies prime but in the first case we already know much about χ mod `. Heuristically, we expect
Elkies and Atkin primes to represent both about 50% of all primes, but we cannot invoke the
Chebotarev density theorem since we do not work in a fixed number field. Under this heuristic,
by considering only Elkies primes, we expect the largest ` to be in O(loge q). Therefore, the SEA
4
algorithm has a heuristic complexity of O(log q). However, although this heuristic complexity
e
is backed by numerical experiments, Satoh and Galbraith showed in [125, Appendix A] that
under GRH, the largest ` to consider in the SEA algorithm is in O((log q)2+ε ).
In order to extend these improvements to point-counting in genus 2, analogues of modular
polynomials were introduced in [60] along with an algorithm to compute them and experiments
on their factorization patterns. Unfortunalety, the complexity estimate to obtain these polyno-
mials is in O(`8 log q) bit operations, which is more costly than the natural extension of Schoof’s
algorithm. In some favorable cases, i.e. when the curve has RM by a small quadratic order,
Milio and Martindale [103, 98] have computed analogues of modular polynomials which could
be exploited to mimic the Atkin improvement. By computing modular correspondences between
Abelian varieties equipped with a theta-structure, Faugère, Lubicz and Robert propose another
extension of modular polynomials in higher dimension [48]. In order to extend the Elkies im-
provement to the genus-2 case, current work by Couveignes and Ezome and implementations
by Milio [104, 37] involve computing (`, `)-isogenies from their kernels, which solves a part of
the problem, but we still lack an algorithm to compute the kernel itself. We also refer to the
AVIsogenies software [20] for ongoing work in that direction although it requires hypotheses on
the rationality of 2- and 4-torsion, and therefore in most cases to accept working in a significant
extension of the base field.
 Part II

Contributions

51
 Chapter 4

Cantor’s division polynomials

As explained in Chapter 1, we compute a low-degree representation of the Frobenius endomor-

phism by successive squarings and reductions in the `-torsion ideal. To make these reductions
possible, we compute the equations of the torsion ideal by formally equating `D = 0 for D a
generic divisor and put them in a “nice” form by solving a polynomial system derived from
`D = 0. This has a complexity cost which has to be controlled, and which depends on param-
eters such as the degree of the ideal, the number of variables and the degrees of the equations,
as detailed in Chapter 2.

Given a generic point P = (x, y) on an imaginary hyperelliptic curve, we show in this

chapter that the divisor `P has coordinates that are rational fractions in x and y and bound
their degrees. This is used in Chapters 3 to 6 to bound the degrees of polynomial systems
involved in the modelling of the torsion subgroups.

These hyperelliptic counterparts to division polynomials were first described by Cantor

in [28], although an alternative strategy was suggested at the same time in Kampkötter’s the-
sis [77] to compute scalar multiplications. Cantor’s paper is quite long and technical, so we do not
attempt to make this chapter self-contained and advise the meticulous reader to keep it handy
while browsing through our proofs. Still, in Section 4.1 we give details on how Cantor’s paper
works as well as some intuition behind the objects that we define to make this chapter as under-
standable as possible without previous knowledge of Cantor’s polynomials. More precisely, we
have tried to make this whole chapter self contained for the reader willing to accept statements
from [28] without proof but unwilling to read the paper. Lastly, we emphasize that this section
is purely technical and can easily be skipped without jeopardizing the reader’s understanding of
this thesis, as we only reuse the main statements proven in Sections 4.2 and 4.3.

While the description of `P using 2g + 2 polynomials was established in Cantor’s original

paper [28], the degrees of only two of these polynomials were actually computed, whereas the
polynomial systems of Chapters 3 to 6 involve all the 2g + 2 polynomials. In genus 2, the degrees
of all these polynomials were computed precisely as seen in Proposition 3.1 but no result was
published in larger genus, although numerical evidence suggests that the degrees are quadratic in
`. In Section 4.2, we prove a bound in Og (`3 ) for the degrees of Cantor’s `-division polynomials
in arbitrary genus. In Section 4.3, we prove that in genus 3, Cantor’s polynomials have degrees
in O(`2 ). These two sections are joint work with Pierrick Gaudry and Pierre-Jean Spaenlehauer
and are to appear as [1, Sec. 6] and [2, Sec. 6].

53
54 Chapter 4. Cantor’s division polynomials

4.1 Overview on division polynomials

As seen in Section 1.1.4, given a point of coordinates (x, y) on an elliptic curve and an integer
` > 1, explicit formulas for the coordinates of the point `P have been known for long and can
be described using the so-called division polynomials ψ` . Recall that these polynomials follow a
recurrence formula which we restate in a different form:
!
ψs−1 ψr ψs ψr−1
∀ s ≥ r ≥ 1, ψs−r ψs+r = det ,
ψs ψr−1 ψs+1 ψr

and that deg ψ` = (`2 − 1)/2.

We can already illustrate why those degrees are important: in Schoof’s algorithm they
determine the size of the quotient ring in which we compute the Frobenius, and therefore the
cost of each operation.

Theorem 4.1 ([28], Th. 8.35). Let C be a hyperelliptic curve given of genus g by an equation
of the form Y 2 = F (X) with F monic of degree 2g + 1. Let P be the generic point on C, (x, y)
be its coordinates and let D = P − P∞ be the associated divisor.
For ` ≥ g, there exists two polynomials δ` (X) and ε` (X) of respective degrees g and g − 1
such that the non-normalized Mumford form of `D is

x−X x−X
    
δ` , ε` .
4y 2 4y 2

Furthermore, the coefficients of δ` are polynomials in x. And those of ε` /y are rational fractions
whose numerators and common denominator are also polynomials in x.

By non-normalized Mumford form, we mean that the polynomial δ` is not monic, contrary
to Definition 1.29. This is the only difference and it allows us to have polynomials as coefficients
of δ` .

Definition 4.2. Let ` ≥ g, the g + 1 coefficients of the polynomials δ` , the g numerators and
the common denominator of the coefficients of ε` /y are called Cantor’s `-division polynomials,
and we omit the ` when there is no ambiguity on it.

In this chapter, we study the degrees in x of these polynomials, and notably their dependency
in `. For a polynomial P whose coefficients are rational fractions, we denote by degmax(P ) the
maximum of the degrees of the numerators and denominators of its coefficients. In the remainder
of the chapter, we aim to bound degmax(δ` ) and degmax(ε` /y).

Warning: Instead of the coefficients of δ` , we may also consider those of δ` ((x − X)/(4y 2 )) or
more often the 2g + 2 polynomials (di )0≤i≤g and (ei )0≤i≤g such that
* g−1 g−1 +
g
X di (x) i X ei (x) i
`D = X + X ,y X .
i=0
dg (x) e (x)
i=0 g

The second family of polynomials is deduced from the first after developping (x − X)/(4y 2 ), and
the third comes from the second after simplifying the rational fractions. For simplicity, all of
them are called division polynomials, but there is little ambiguity on their respective occurrences:
the last one is the only form appearing in our systems and in practice, while we mostly focus on
4.1. Overview on division polynomials 55

the first one when proving bounds on degrees. However, the difference of degrees between them
only depends on g and can readily be computed.
To simplify the exposition, the first step is a change of variable X = x − z from the point
P = (x, y) on the curve Y 2 = F (X) to the point P0 = (0, (−1)g+1 y) on the curve C 0 of equation
Y 02 = E(z) with E(z) = F (x − z). The choice of the sign of the ordinate of P0 is well-motivated
in [28], but since we ultimatelyponly focus on the degrees of Cantor’s polynomials, we prefer
not to linger on signs. Denote E(z) the formal power series which is the Taylor series of the
square root of E around z = 0 with constant term (−1)g+1 y. Following Cantor, we first define
unnormalized division polynomials as A` , B` , C` and D` , then we normalize them by the right
power of 2y and we invert the change of variables to recover the normalized polynomials α` , β` ,
γ` and δ` . Lastly, the polynomial ε` is deduced from δ` , δ`−1 and δ`+1 .
Let us now consider the curve C 0 , mapped in its Jacobian J 0 by P 7→ P − P∞ . Let A` (z) and
B` (z) be polynomials such that

• z ` divides A` (z) − B` (z) E(z),

p

• 2 deg A` ≤ ` + g and 2 deg B` + 2g + 1 ≤ ` + g.

We are not sure yet whether they exist and how to compute them, but this will be dealt with
once their definition becomes more natural and relevant to the initial problem.
Indeed, the function on the curve C 0 given by A` (z) − Y 0 B` (z) has ` + h poles at infinity with
h ≤ g. Then, denote D the associated principal divisor, we have D = D0 + `P0 − (` + h)P∞
where D0 is an effective degree-h divisor, since z ` divides A` (z) − Y 0 B` (z). Now, this principal
divisor has to be zero in the Jacobian J 0 , so we end up with D0 − hP∞ = −`(P0 − P∞ ).
For ` > g define
D` (z) = −(A` (z)2 − B` (z)2 E(z))/z ` (4.1)

as in [28, 2.3]. This definition is natural in the sense that D` is the Mumford u-coordinate of
D0 . Then, we define Ē` (z) to be the corresponding v-coordinate of the Mumford form, that is
deg Ē` < deg D` and Ē` (z)2 − E(z) ≡ 0 mod D` . This gives the intuition on the construction of
the non-normalized division polynomials, but deeper understanding is required to define them
rigorously, which actually comes with the existence and definition of A` and B` .
The first condition on these polynomials amounts to ` homogeneous linear conditions on their
(unknown) coefficients. The degree conditions only allows for deg A` + deg B` to be `−1 so there
is a total of exactly `+1 coefficients to be determined to completely fix those two polynomials.
p In
other terms, A` and B` are defined as Padé-Hermite approximants of the series E(z) modulo
z ` . Thus, by unicity of the Padé-Hermite approximants, either there is no solution, or there
is a unique solution for A` and B` up to multiplication by a scalar. There exist algorithms to
compute these Padé approximants, and the condition for their existence is the non-nullity of
some Hankel determinants.
p
For brevity, let us define the power series S(z) = E(z), denote sj being either the j-th
coefficient of S or 0 if j ≤ 0. For m ≥ 0 and n ≥ 1 let us define the following Hankel matrix as
 
sm−n+1 sm−n+2 ··· sm
s
 m−n+2 sm−n+3 ··· sm+1 


Hmn (S) =  .. .. .. .. 
,
 . . . . 
 sm−1 sm ··· sm+n−2 
 

sm sm+1 ··· sm+n−1

56 Chapter 4. Cantor’s division polynomials

and hmn (S) its determinant with the convention hmn (S) = 1 if n = 0 and hmn (S) = 0 if n ≤ −1.
The non-nullity of these hmn guarantees the existence of solutions to the Padé approximation
problem, as stated in [28, Th. (3.5)]. These solutions are actually (up to a constant) determinants
of matrices similar to Hmn . We do not restate them since that would not be enlightening, but
properties of Padé approximants allow to define the polynomials A` (z), B` (z) as well as two
other quantities which will play a crucial role: the series C` (z) such that
A` (z) − B` (z)S(z) = −z ` C` (z),
and the f` which are polynomials in x, defined as hn`+1 m`+1 where n` and m` are some indices
depending on ` and g which we do not detail. We will see later that these f` are actually
non normalized versions of polynomials ψ` that extend the elliptic division polynomials in some
natural way [28, Cor. 8.34].
Using properties of Padé approximants called Frobenius identities in [74, Eq. (2.5)], one can
derive the following recurrence formulas:
Proposition 4.3 ([28], 3.14). For ` ≥ g + 1,
f`−1 A`+1 (z) = f` A` (z) − zf`+1 A`−1 (z),
f`−1 B`+1 (z) = f` B` (z) − zf`+1 B`−1 (z),
f`−1 C`+1 (z) = (f` C` (z) − f`+1 C`−1 (z))/z.
Along with initial values given in [28, 3.10], these identities allow to compute the A, B, C and
f inductively without having to compute the determinants. More importantly for our purpose,
they allow to inductively bound degmax(A), degmax(B) and degmax(C) once the degrees of
the fi are known. In Section 4.2, we transcript them into recurrence relations involving the
normalized counterparts α, β, γ and ψ of Definition 4.4 instead of directly studying the non
normalized objects. This is done for a pragmatic reason: to avoid duplicating proofs and results
in Cantor’s paper, the non-normalized objects are used to simplify technical proofs but final
results are only given in normalized form.
Let us introduce some notation: for S(z) a formal power series, we denote S[ιn ] the polyno-
mial of degree ≤ n obtained by truncation, i.e. nk=0 sk z k and S[n] the n-th term of the series,
P

that is sn z n .
We restate the dictionary to switch from the non-normalized to the normalized world.
Definition 4.4 ([28], 8.7). Let ` ≥ g + 1 and ν` = (`2 − ` − g 2 + g)/2, we define
ψ` = (2y)ν` f` ,
α` (z) = 2(2y)ν`−1 −1 f` A` (4y 2 z){ιg },
β` (z) = (2y)ν`−1 f` B` (4y 2 z){ιg },
γ` (z) = (2y)ν`+1 f` C` (4y 2 z){ιg },
δ` (z) = (2y)2ν` D` (4y 2 z),
ε` (z) = Ē` (4y 2 z).
From the non-normalized conditions of the Mumford form, we get the following alternative
expression for ε` , which allows us to focus on ψ` and δ` (z).
Proposition 4.5 ([28], 8.13). For ` > g,
2 δ 2 (z) − ψ 2 δ
z(ψ`−1 `+1 `+1 `−1 (z))
ε` (z) = y 2 mod δ` (z).
ψ`−1 ψ` ψ`+1
4.1. Overview on division polynomials 57

The five polynomials α to ε have degrees at most g in z, but their coefficients are a priori
rational fractions in x and y. The following theorem clarifies the situation:
Theorem 4.6 ([28], 8.15). If ` − g is even, then ψ` is a polynomial in x, and α` (z)/(2y)g ,
β` (z)/(2y)g and γ` (z)/(2y)g are polynomials in z with coefficients that are polynomials in x.
If ` − g is odd, then ψ` /(2y)g is a polynomial in x, and α` (z), β` (z) and γ` (z) are polynomials
in z with coefficients that are polynomials in x.
Definition 4.7 ([28], 8.16). Let us now define P` as ψ` if ` − g is even, and ψ` /(2y)g otherwise,
so that P` is always a polynomial in x.
Both its degree and leading coefficient are given but we only restate the result on the degree.
Theorem 4.8 ([28], 8.17). The polynomial P` has degree
g(`2 − g 2 )

if ` − g is even,


2

2 2
 g(` − g ) − g(2g + 1)

if ` − g is odd.

2
Using the fact that y 2 = F (x), we can rephrase this by a formula which we often use to prove
the following theorems: deg(ψ`2 ) = g(`2 − g 2 ). We now have all the necessary ingredients for
Section 4.2 but we present alternative recurrence formulas that are more similar to the elliptic
case, and that also allow for sharper bounds in the genus-3 case.
This is studied in [28, Sec. 6 & 8], respectively in the non-normalized and normalized cases.
Once again, the idea is to use properties of Padé approximants and translate them into recurrence
relations. We focus on the results and refer to Cantor’s paper for proofs, hence we restrict to
the normalized case. We restate the relations [28, 8.31 to 8.33] that allow to express α` , γ` and
ψ` in terms of determinants involving the polynomials αr , γr and ψr for several values of r that
are close to `/2.
Definition 4.9 ([28], 8.30). Let s ≥ r ≥ 2g − 1 and h ≤ g, define the (g + 1) × (g + 1) matrix
 
αr−g (z)αs (z)[ιg−2 ] ψr−g ψs γr−s (z)γs (z)[h]
αr−g+1 (z)αs−1 (z)[ιg−2 ] ψr−g+1 ψs−1 γr−g+1 (z)γs−1 (z)[h]
 
Ers [h] =  .. .. .. .
. . .
 
 
αr (z)αs−g (z)[ιg−2 ] ψr ψs−g γr (z)γs−g (z)[h]
Definition 4.10 ([28], 8.32). Let s ≥ r ≥ 2g − 1 and h ≤ g, define the (g + 1) × (g + 1) matrix
 
αr−g (z)αs (z)[h] ψr−g ψs γr−s (z)γs (z)[ιg−2 ]

α
 r−g+1 (z)α s−1 (z)[h] ψ ψ
r−g+1 s−1 γ r−g+1 (z)γs−1 (z)[ιg−2 ]

Frs [h] = 
 .. .. .. .
. . .

 
αr (z)αs−g (z)[h] ψr ψs−g γr (z)γs−g (z)[ιg−2 ]
Recall that, using previous notation, the first matrix involves g − 1 terms in α and one term
in γ while the second one involves g − 1 terms in γ and one in α. We restate recurrence relations
based on the determinants of these matrices:
Proposition 4.11 ([28], 8.31). For s ≥ r ≥ 2g − 1 and h ≤ g, we have
g−1
g+1
det Ers [h] = (−1)( 2 ) γr+s−2g+1 [h]ψs−r
Y
ψr−g+k ψs−g+k .
k=1
58 Chapter 4. Cantor’s division polynomials

Proposition 4.12 ([28], 8.33). For s ≥ r ≥ 2g − 1 and h ≤ g, we have

g−1
g+1
det Frs [h] = (−1)( 2 ) αr+s−2g+1 [h]ψs−r
Y
ψr−g+k ψs−g+k .
k=1

When g = 2, and for s ≥ r ≥ 3 these formulas yield

 
ψs−2 ψr ψs−1 ψr+1 ψs ψr+2
ψs ψr ψs+r ψs−r = det ψs−1 ψr+1 ψs ψr ψs+1 ψr+1  .
 
ψs ψr+2 ψs+1 ψr−1 ψs+2 ψr

This can be used to compute the exact degrees of Cantor’s division polynomials in genus 2,
which is used in Chapter 3. Another important remark is that when g = 1 they give exactly the
same recurrence as the one satisfied by the division polynomials, and the immediate corollary is
that in genus 1 the ψ` coincide with the previously known division polynomials.
To our knowledge, apart from the leading and constant coefficients of δ` , which Cantor proved
to be respectively −(4y 2 )g ψ`2 and (−1)g+1 ψ`−1 ψ`+1 even in arbitrary genus, no additional proven
result was published for g ≥ 3. In Section 4.3, we instantiate these recurrence formulas in genus
3 and use them to prove a quadratic bound in ` on the degrees of all the 8 analogues of division
polynomials. We also explain why we are not very optimistic about that approach compared to
the first one when g is larger.

4.2 A cubic bound in any genus

We prove the following:

Theorem 4.13. For any integer ` > g, the polynomial δ` (X) of degree g in X has coefficients in
Fq [x] whose degrees in x are bounded by g`3 /3 + Og (`2 ); the polynomial ε` (X)/y has coefficients
in Fq (x) such that the degrees of the numerators and the denominators have degrees bounded by
2g`3 /3 + Og (`2 ). Furthermore, the roots of the denominators are roots of the leading coefficient
of δ` (X).

Proof. Technicalities arise from the normalizations required to manipulate entities that are poly-
nomials in x (and not rational fractions), without odd power of y involved. In Cantor’s article,
this normalization often depends on the parity of ` − g. We will concentrate on the case where
g is even; for the other case some formulas must be adapted, multiplying or dividing by 2y at
various places.
We recall that ν` = (`2 − ` − g 2 + g)/2 as defined in (8.7), so that ν` = ν`−1 + ` − 1. By
combining Definition 4.4 and 4.1, we obtain

(2y)2ν`  2 2 2 2 2

δ` (z) = A ` (4y z) − B` (4y z) E(4y z) ,
(4y 2 z)`

where A` and B` are unnormalized versions of α` and β` given in Definition 4.4 and E(z) is
defined by E(z) = f (x − z). For our purpose, it is easier to deal with non-truncated versions of
α` and β` . Let us then introduce the following quantities, inspired by Definition 4.4:

ᾱ` (z) = 2(2y)ν`−1 −1 A` (4y 2 z), and β̄` (z) = (2y)ν`−1 B` (4y 2 z),
4.2. A cubic bound in any genus 59

so that δ` can be rewritten as

1 1
 
δ` (z) = `
ᾱ` (z)2 − 2 β̄` (z)2 E(4y 2 z) .
4z y
By Theorem 4.6, the coefficients of α` (z) and β` (z) are polynomials in Fq [x], and the proof is
also valid for the non-truncated versions ᾱ` (z) and β̄` (z). Note that here we use the fact that g
is even, so that the potential adjusting factor (2y)g is an even power of y that can be rewritten
in terms of F (x). The polynomial E(4y 2 z) has coefficients which are polynomials in x of degree
bounded by (2g + 1)2 . Therefore, in order to obtain a degree bound for the coefficients of δ` (z),
it is sufficient to bound the coefficients of ᾱ` (z) and β̄` (z).
We are interested in a bound for fixed genus g and when ` grows to infinity and we use the
Og notation as a O notation which also hides factor depending only on g (and not on `). For k
in [1, `], we will use an induction to bound degmax(ᾱk (z)) and degmax(β̄k (z)). For k ≤ g + 1,
none of these quantities depends on `, so that all the degrees can be bounded by an expression
in g only, i.e. in Og (1). For k ≥ g + 1, we start from Proposition 4.3 where we substitute k for
`, we evaluate it at 4y 2 z, and we multiply by (2y)2νk −1 , so that we obtain:

(2y)νk fk−1 ᾱk+1 (z) = (2y)νk +k−1 fk ᾱk (z) − (2y)νk +2k−1 fk+1 z ᾱk−1 (z),

where all the polynomials have coefficients in Fq [x]. The expression for β̄k is exactly the same, but
we have to multiply the expression in Proposition 4.3 by (2y)2νk in that case. By Definitions 4.4
and 4.7 plus Theorems 4.6 and 4.8, for any k, the quantity (2y)νk fk is a polynomial in x of degree
g(k 2 −g 2 )/2. Therefore the right-hand-side of the recurrence relation has coefficients with degrees
bounded by an expression of the form max degmax(ᾱk (z)), degmax(ᾱk−1 (z)) + gk 2 /2, up to

a term linear in k and cubic in g. We finally get:

degmax(ᾱk+1 (z)) ≤ max degmax(ᾱk (z)), degmax(ᾱk−1 (z)) + gk 2 /2 + Errg (k),

where Errg (k) is a polynomial linear in k and cubic in g. Again, this inequality is also valid for
β̄k . By induction, we then get the following bounds:

g`3 g`3
degmax(ᾱ` (z)) ≤ + Og (`2 ), and degmax(β̄` (z)) ≤ + Og (`2 ).
6 6
We can then propagate these bounds in the expression of δ` and we get
degmax(δ` (z)) ≤ max(2 degmax(ᾱ` (z)), 2 degmax(β̄` (z)) + degmax(E(4y 2 z)), so that we get the
claimed result concerning δ` .

The fact that ε` (z)/y has coefficients in Fq (x) follows directly from Proposition 4.5 that we
recall here:  
2 δ 2
z ψ`−1 `+1 (z) − ψ`+1 δ`−1 (z)
ε` (z) = y mod δ` (z).
ψ`−1 ψ`2 ψ`+1
As stated in [28, 8.11], the leading coefficient of δ` (z) is −(4y 2 )g ψ`2 , so that the property on the
denominator of ε` can not be easily deduced from this equation, due to the presence of ψ`−1 and
ψ`+1 before the reduction modulo δ` (z) occurs. We will prove it below, with a direct geometric
argument, but we first give bounds on the degrees of the coefficients of the numerator and the
denominator.
The polynomial δ` (z) is of degree g in z, so that at most two steps of reduction are required to
reduce the degree of ε` to strictly less than g. In fact, it can be checked that LT(ψ`−1 2 δ
`+1 (z)) =
60 Chapter 4. Cantor’s division polynomials

LT(ψ`+12 δ
`−1 (z)), so that there is at most only one reduction step. This reduction accounts for an
increase of the coefficients’ degrees in x by at most degmax(δ` ) in the numerator and an increase
of the degree of the leading coefficient of δ` in the denominator. Since deg ψ` = g`2 /2 + Og (`),
the degrees of the coefficients of the numerator of ε` (z) are bounded by 32 g`3 + Og (`2 ), and the
degree of the denominator is bounded by 3g`2 + Og (`).
It remains to prove the claim on the roots of the denominator of the coefficients of ε` (z)/y.
For this, we consider the map from the affine part of the curve Caff to J seen as a projective
Abelian variety, that sends a point (x, y) to [`]((x, y) − ∞). One of the main points of Cantor’s
article is that if ψ` (x) 6= 0, then the image by this map is in J \ Θ, where Θ ⊂ J is the subvariety
of elements of weight less than g (i.e. divisors that are sums of less than g points). On this open
subset, Mumford coordinates with a monic u of degree g and v of degree at most g − 1 give a
local set of coordinates that we use to describe the map. The i-th coefficient of v is y times a
rational fraction ci in x that gives a finite value at any x for which ψ` (x) 6= 0. Therefore, any
root of the denominator of ci is a root of ψ` . By Theorem 4.1, the Mumford v-polynomial that
we are considering is ε` up to a renormalization that will only introduce additional powers of
4y 2 in the denominator. Therefore, any root of the denominator of the coefficients of ε` is a root
of ψ` or of 4y 2 , and both divide the leading coefficient of δ` , which is −(4y 2 )g ψ`2 .

Remark. The bounds that we obtain are not tight: from [28], we know that the leading
and constant coefficients are in Og (`2 ) instead of Og (`3 ). We ran experiments that allow us to
conjecture
  precise degrees
 for the other coefficients. In these experiments, instead of developing
x−X x−X
δ` 4y2 and ε` 4y2 to compute the di ’s and ei ’s, we computed `((x, y)−∞) over the function
field of the curve. This does not exactly yield the di ’s and ei ’s because we actually get di /dg and
ei /eg , thus possibly missing a common factor in all the di ’s and ei ’s. We denote d˜i and ẽi the
numerators and denominators of the aforementioned fractions, and we compute their degrees
for each pair (g, `) with g ≤ 8 and g < ` ≤ g + 20 (which includes non prime values of `). We
found that the degrees of the d˜i are consecutive from deg(d˜g ) up to deg(d˜0 ) = deg(d˜g ) + g, with
the following values for deg(d˜0 ).

g`2 − g 3 + g
(
if g − ` is even
2 3 2
g` − g + 2g − 1 if g − ` is odd
Concerning the ẽi , the degrees are consecutive from deg(ẽg−1 ) up to deg(ẽ0 ) = deg(ẽg ), the
latter being equal to

3(g`2 − g 3 )/2 + 2g 2 − g − 1
(
if g − ` is even
3(g`2 − g 3 )/2 + 3g 2 − g/2 − 1 if g − ` is odd

Cantor [28] gave simple expressions for the leading term and constant term of δ` (respec-
tively −(4y 2 )g ψ`2 and (−1)g+1 ψ`−1 ψ`+1 ), from which we can deduce the degrees of d0 and dg by
evaluating δ` at (x − X)/4y 2 . Assuming that there is no common factor to all the di ’s when
g − ` is even, while the GCD of all the di ’s is f g−1 when g − ` is odd, these theoretical degrees
are consistent with our experiments.

4.3 A quadratic bound in genus 3

The previous cubic bound was sufficient because its error only affects the final complexity bound
by a constant in some O(). In genus 3, however, we want to compute the exact exponent of log q
4.3. A quadratic bound in genus 3 61

in the complexity so we need a bound that is quadratic in `. We do so by using other induction

formulas.

Theorem 4.14. In genus 3, the degrees of Cantor’s `-division polynomials are bounded by O(`2 ).

We first prove a bound on the degrees of the coefficients of the quantities αr and γr defined
in [28], from which the wanted bounds will follow. The key tools are Propositions 4.11 and 4.12
that relate quantities at index ` to quantities at index around `/2, in a similar fashion as for the
division polynomials of elliptic curves. More precisely, the following lemma shows that when
the index ` is (roughly) doubled, degmax α` and degmax γ` are roughly multiplied by 4, which
leads to the expected quadratic growth.

Lemma 4.15. Let ` ≥ 10, and assume that for all i ≤ (` + 9)/2 the degrees degmax αi and
degmax γi are bounded by C, then degmax α` and degmax γ` are bounded by 4C + 36` + 108.

Proof. We first deal with the bound on degmax γ` . Let us consider r and s around `/2 such that
` = r + s − 5: we take either r = s − 3 = `/2 + 1 if ` is even, or r = s − 4 = (` + 1)/2 otherwise.
From Definition 4.9 and Proposition 4.11, the degree of γ` [h]ψs−r ψr−2 ψs−2 ψr−1 ψs−1 is that
of the determinant of the matrix Ers [h] defined by:
 
αr−3 αs [0] αr−3 αs [1] ψr−3 ψs γr−3 γs [h]
α α α α ψ ψ γ
[0] [1] r−2 γs−1 [h]

Ers [h] =  r−2 s−1 r−2 s−1 r−2 s−1
.

αr−1 αs−2 [0] αr−1 αs−2 [1] ψr−1 ψs−2 γr−1 γs−2 [h]
αr αs−3 [0] αr αs−3 [1] ψr ψs−3 γr γs−3 [h]

Therefore we have an expression for the degrees of the coefficients of γ` in terms of objects at
index around r and s:

deg γ` [h] ≤ deg det Ers [h] − deg(ψr−2 ψs−2 ψr−1 ψs−1 ).

In this last formula, the factor ψs−r has been omitted, because s − r is either 3 or 4, and by
Theorem 4.8 this has non-negative degree in any case. Thus, we simply bounded it below by
0 in the previous inequality. Before entering a more detailed analysis, we use the fact that
αk (0) = ψk−1 and γk (0) = ψk+1 (this is [28, (8.8)]) to rewrite the first column with expressions
for which we have exact formulas for the degree:
 
ψr−4 ψs−1 αr−3 αs [1] ψr−3 ψs γr−3 γs [h]
ψ ψ α α ψ ψ γ
[1] r−2 γs−1 [h]

Ers [h] =  r−3 s−2 r−2 s−1 r−2 s−1
.

ψr−2 ψs−3 αr−1 αs−2 [1] ψr−1 ψs−2 γr−1 γs−2 [h]
ψr−1 ψs−4 αr αs−3 [1] ψr ψs−3 γr γs−3 [h]

The determinant of Ers [h] is the sum of products of 4 ψ factors and 4 α or γ factors. The
degrees of the former are explicitly known, while by hypothesis we have upper bounds on the
latter, since all the indices are at most (` + 9)/2. We can then deduce an upper bound on the
degree of this determinant. All the ψi have indices with i in the range [r − 4, s] (remember that
r ≤ s), and since their degrees increase with the indices, we can upper bound the degree of the
products of the four ψ factors by 4 deg ψs . Therefore we have

deg det Ers [h] ≤ 4(deg ψs + C).

62 Chapter 4. Cantor’s division polynomials

In order to deduce an upper bound on degmax γ` , it remains to get a lower bound on the degree
of the deg(ψr−2 ψs−2 ψr−1 ψs−1 ) term, and again by monotonicity of the degree in the index, it is
bounded below by 4 deg ψr−2 . So finally, we get

degmax γ` ≤ 4C + (deg ψs4 − deg ψr−2

4
).

Using Definition 4.7 and Theorem 4.8, we deduce that for all k, we have deg(ψk2 ) = 3(k 2 − 9)
and substituting this value and the expression of r − 2 and s in term of `, we obtain
(
30` + 90 if ` is even,
deg ψs4 − deg ψr−2
4
=
36` + 108 if ` is odd,

and the result follows for degmax γ` .

The proof for degmax α` follows the same line. Using the matrix Frs [h] of Definition 4.10
in a similar way as we used the matrix Ers [h] and with the help of Proposition 4.12, we end up
with the following bounds
(
4C + 30` − 30 if ` is even,
degmax α` ≤
4C + 36` − 36 if ` is odd,

which are stricter than our target.

Finally, the bound ` ≥ 10 is sufficient to ensure that the quantities r and s are at least 5, as
required to apply Propositions 4.11 and 4.12. This condition would still hold for ` as small as
8 but our recurrence needs ` > 9 to propagate, or else (` + 9)/2 would be greater or equal to
`.

We can now finish the proof of Theorem 4.14. We define two sequences (`i )i≥0 and (Ci )i≥0
as follows: let `0 = 10 and let C0 be a bound on the degrees of the coefficients of all the αi and
γi for i ≤ `0 . Then for all i ≥ 1, we define the sequences inductively by
(
`i+1 = 2`i − 9
Ci+1 = 4Ci + 36`i+1 + 108.

By Lemma 4.15, for all i, and all ` ≤ `i , the degrees degmax α` and degmax γ` are bounded by
Ci . The expression `i = (`0 − 9)2i + 9 = 3 · 2i + 9 can be derived directly from the definition
and substituted in the recurrence formula of Ci+1 to get Ci+1 = 4Ci + 216 · 2i + 432. This
recurrence can be solved by setting Γi = Ci + 108 · 2i + 144, so that Γi+1 = 4 Γi , and we obtain
Ci = (C0 + 252) 4i − 108 · 2i − 144. Finally, for any `, we select the smallest i such that ` ≤ `i .
This value of i is dlog2 ((` − 9)/3)e. The corresponding bound for degmax α` and degmax γ`
is then Ci , which grows like O(`2 ) (and we remark that the effect of the ceiling can make the
constant hidden in the O() expression grow by a factor at most 3).
Using [28, Eq. 8.10], i.e. δ` (z) = α` (z)γ` (z){ιg }, we have degmax δ` ≤ degmax α` +degmax γ` ,
and therefore the bound O(`2 ) also applies to the degrees of the coefficients of δ` . And by
Proposition 4.5, the same holds as well for the coefficients of ` /y.
This concludes the proof of Theorem 4.14.

Remark. One could try to extend this method to larger g in the hope of getting a better
bound than the cubic one proven in the previous section. In a nutshell, the quadratic bound
was achieved because the 4 × 4 determinants involve 2 terms in either α2 or γ 2 and 2 terms in
4.3. A quadratic bound in genus 3 63

ψ 2 . This led to a bound on the degrees of α and γ that was mutiplied by 4 each time ` was
multiplied by 2. In larger genus, however, the balance between the two types of terms is broken
because the (g + 1) × (g + 1) determinant is made up of (g − 1) terms in α2 and β 2 . A direct
generalization of our method would therefore give a bound B` on the degrees of α and β that is
multiplied by 2(g − 1) each time ` is multiplied by 2. In particular, for g ≥ 5, the growth of B`
seems already worse than cubic.
64 Chapter 4. Cantor’s division polynomials
 Chapter 5

Asymptotic complexity bounds in

arbitrary genus

Let C be a hyperelliptic curve of genus g over a finite field Fq of characteristic p and denote
by J its Jacobian. In this chapter, we present a Las Vegas algorithm derived from Schoof and
Pila’s approaches to count points on hyperelliptic curves that achieves a time complexity in
O((log q)cg ) for c a constant and g fixed with q growing and p large enough. This is joint work
with Pierrick Gaudry and Pierre-Jean Spaenlehauer and most of this chapter is to appear as [1].

Organization of the chapter. Section 5.1 describes a general algorithm for point-counting
on Abelian varieties along with its complexity, assuming that the `-torsion can be efficiently
computed. Section 5.2 establishes the complexity result for multi-homogeneous polynomial
systems that is required to obtain our claimed complexity bound. Section 5.3 contains the
modelling of the `-torsion under some mild assumptions on its structure. Finally, Section 5.4
describes the complete modelling of the `-torsion, which is faithful even if the assumptions
required in Section 5.3 are not satisfied.

5.1 Overview
This chapter aims to give a proof of the following result:

Theorem 5.1. There exists an explicitly computable constant c such that for all genus g, there
exists an integer q0 (g) such that for all prime power q = pn larger than q0 (g) with p ≥ (log q)cg
and for all imaginary hyperelliptic curves C of genus g defined over Fq , the numerator L of the
local zeta function of C from Proposition 1.42 can be computed with a probabilistic algorithm in
expected time bounded by (log q)cg .

This complexity result is summarized by the notation Og ((log q)O(g) ), keeping in mind that
g is fixed and q grows to infinity. Indeed, such a complexity statement can hide any factor that
0
depends only on g: a running time in f (g)(log q)cg can be transformed into (log q)c g by taking
0
a value c0 larger than c and adjusting q0 (g), so that |f (g)| ≤ (log q0 (g))(c −c)g .
A typical example is the multiplication of two polynomials of degree d = (log q)O(g) . Us-
ing FFT-based techniques, this can be done in O(d) e operations, which can be rewritten as
(log q) O(g) (log((log q) O(g) )) for some constant k and is therefore again in Og (log(q)O(g) ). Here
k

the function f (g) that has been hidden in the operation is polynomial in g, but we will have

65
66 Chapter 5. Asymptotic complexity bounds in arbitrary genus

cases where it is a combinatorial factor that grows very quickly with g and we make no effort
to optimize it.
The algorithm that allows to prove the theorem is essentially the same as the one proposed
by Pila for Abelian varieties, which is itself inspired by Schoof’s algorithm for counting points
on elliptic curves. Pila’s algorithm reconstructs the numerator of the local zeta function of C by
computing the action of the Frobenius on the `-torsion for sufficiently-many prime numbers `
and by using the Chinese Remainder Theorem. A bird’s eye view of this algorithm is given in
Algorithm 6. The main difficulty resides in the step where one computes an explicit description
of J[`]. Since J[`] is a 0-dimensional variety of degree `2g , what we will compute is a geo-
metric resolution of the corresponding radical ideal, that is a univariate squarefree polynomial
F` (T ), together with 2g coordinate polynomials γi (T ), such that the coordinates of the `-torsion
elements are the evaluations of the vector (γ1 (T ), . . . , γ2g (T )) at the roots of F` .

input : q ∈ Z>0 a prime power, and f ∈ Fq [X] a monic squarefree of degree 2g + 1.

output: The characteristic polynomial χ ∈ Z[T ] of the Frobenius endomorphism on the
Jacobian J of the hyperelliptic curve defined over Fq with Weierstrass form
Y 2 = f (X).
` ← 1;
R ← 1;
while R ≤ 2 2g

g
g q + 1 do
` ←NextPrime(`);
if ` divides q then
` ←NextPrime(`);
end
Compute a description of J[`];
Compute a 2g × 2g matrix F with coefficients in Z /` Z representing the action of
the Frobenius on J[`] ∼
= (Z /` Z)2g ;
Compute the characteristic polynomial χ mod ` of the matrix F ;
R ← R · `;
end
Reconstruct χ using the Chinese Remainder Theorem.
Algorithm 6: A bird’s eye view of Pila’s point counting algorithm for hyperelliptic
curves.

To be more precise, the Mumford coordinates are in fact a set of g affine systems of co-
ordinates, each corresponding to a different weight of the represented divisor (the definition is
recalled in Section 5.3). The variety J[`] will accordingly be represented by a set of g geometric
resolutions, each encoding `-torsion divisors of a given weight w ∈ [1, g]. Generically, we expect
that all the elements in J[`] have weight g, except for the neutral element which has weight
0. Most of the chapter is dedicated to computing efficiently this representation for J[`]. The
cornerstone of the proof of Theorem 5.1 relies on the following statement.

Proposition 5.2. Let C be a hyperelliptic curve of genus g over Fq with Weierstrass form
Y 2 = f (X) (f monic squarefree of degree 2g + 1) and J be its Jacobian variety. Let ` > g
be a prime not dividing q. Assuming that the characteristic of Fq is sufficiently large as in
Theorem 5.1, there is a Las Vegas probabilistic algorithm which takes as input q, `, f and which
computes geometric resolutions for the varieties {Jw [`]}w∈[1,g] of `-torsion points of weight w in
the Jacobian variety. This algorithm can be implemented by a Turing machine with space and
5.2. Computing geometric resolutions 67
 
expected time Og (` log q)O(g) .

Assuming this complexity bound, performing a complexity analysis as done in [114] leads
to a complexity bound for Algorithm 6 that corresponds to Theorem 5.1. We recall it here
for completeness, with some simplifications due to the fact that we consider a probabilistic
algorithm, so we can factor polynomials using Cantor-Zassenhaus’ algorithm.

Proof of Theorem 5.1 assuming Proposition 5.2. By Weil’s bounds, the absolute values of the
2g
g
coefficients of the characteristic polynomial χ are bounded by g q . Therefore at the end of
the loop of Algorithm 6, these coefficients are completely determined by their values modulo all
the primes ` that have been explored. It follows from [140, Cor. 10.1] that the largest ` in the
loop is at most linear in g log q. From this and Proposition 5.2, computing the description of J[`]
as a union of geometric
 resolutions for all the Jw [`] can be achieved within expected complexity
Og (log q) O(g) .
Factoring the univariate
 polynomials  involved in the geometric resolutions can be done within
the same time bound Og (log q) O(g) , since the sum of their degrees is `2g and factoring poly-
nomials in finite fields can be done in time linear in log(q) and quasi-quadratic in the degree [54,
Thm. 14.14]. Therefore, it is possible to construct a Mumford representation for each `-torsion
divisor within the same complexity, each of them possibly defined over a different extension of
Fq . In fact, due to the rationality of the group law that acts on J[`], one of these extensions of
Fq contains all the others.
Using elementary linear algebra for the Frobenius endomorphism ϕ acting on J[`] (seen as
an F` -vector space), we can deduce χ` = χ mod `. We first compute a basis of J[`] by brute
force and a dictionary of how all elements decompose on it. Then, the action of ϕ on the basis
elements can be computed and the result is a matrix whose characteristic polynomial is χ` . All
of this fits in the Og ((log q)O(g) ) complexity bound. The loop is repeated Og (log q) times, and
this additional factor does not affect the overall complexity.

5.2 Computing geometric resolutions

5.2.1 Main complexity result
The following proposition is a cornerstone of our complexity result for computing the `-torsion
of the Jacobian of a hyperelliptic curve. The statement and its proof combine three main
ingredients: (1) the geometric resolution algorithm [64] and its version for finite fields [25],
which are methods for solving polynomial systems detailed in Section 2.4 whose complexity
depends mainly on geometric degrees; (2) the multi-homogeneous Bézout bound presented in
Section 2.4.1 which allows us to control the geometric degrees by separating the variables in our
modelling in two blocks, where the block supporting most of the degrees has small cardinality;
(3) a variant of Bertini’s theorem to process our polynomial system into a reduced regular
sequence which is a valid input for the geometric resolution algorithm.
As we shall see in the next sections, our polynomial system modelling the `-torsion will have
two blocks of variables. The first block occurs with large degree `O(1) but it has a very small
cardinality in O(g). The second block has a larger cardinality, but the degrees of the equations
with respect to this block do not depend on `, but only on g. Taking this bi-homogeneous struc-
ture into account is crucial to reach our claimed complexity bound. The following proposition
provides a bound on the complexity of solving polynomial systems having this structure, and
the remainder of this section is dedicated to its proof. This section is devoted to describing
68 Chapter 5. Asymptotic complexity bounds in arbitrary genus

tools that we will use to estimate the complexity of computing a convenient representation of
the `-torsion of the Jacobian of hyperelliptic curves.
Let us recall the notation of Section 2.4.1: if f ∈ Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ], then we
let degx (f ) (resp. degy (f )) denote the degree of f (X1 , . . . , Xnx , y1 , . . . , yny ) ∈ Fq [X1 , . . . , Xnx ]
(resp. f (x1 , . . . , xnx , Y1 , . . . , Yny ) ∈ Fq [Y1 , . . . , Yny ]), where y1 , . . . , yny (resp. x1 , . . . , xnx ) are
generic values in Fq .
Proposition 5.3. There exists a probabilistic Turing machine T which takes as input polynomial
systems with coefficients in a finite field Fq and which satisfies the following property. For any
function h : Z>0 → Z>0 , for any positive number C > 0 and for any ε > 0, there exists
a function ν : Z>0 → Z>0 and a positive number D > 0 such that for all positive integers
g, `, nx , ny , dx , dy , m > 0 such that nx < C g, ny < h(g), dx < h(g) `C , dy < h(g), m < h(g), for
n
any prime power q such that the prime number p dividing q satisfies 2nx +ny dnx x dy y < p, and for
any polynomial system f1 , . . . , fm ∈ Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ] such that
• for all i ∈ [1, m], degx (fi ) ≤ dx and degy (fi ) ≤ dy ,
• the ideal I = hf1 , . . . , fm i has dimension 0 and is radical,
the Turing machine T with input f1 , . . . , fm returns an Fqdν(g) log `e -geometric resolution of the
variety {x ∈ Fq | f1 (x) = · · · = fm (x) = 0} with probability at least 5/6, using space and time
bounded above by ν(g) `D g (log q)2+ε .
Proof. Postponed to Subsection 5.2.3.

5.2.2 Input preparation

Since the geometric resolution requires its input to be a reduced regular sequence, we first need
to ensure that we can construct such a sequence from our input system. A classical way to
achieve this is to replace the input system by a generic linear combination of the polynomials. If
the ideal generated by the input system is 0-dimensional and radical, then a variant of Bertini’s
theorem ensures that the obtained sequence is regular and reduced in the sense of Definitions 2.26
and 2.47.
Proposition 5.4. [132, Thm. A.8.7] Let (f1 , . . . , fm ) ∈ Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ]m be poly-
nomials such that the ideal I = hf1 , . . . , fm i has dimension 0 and is radical. Let dx , dy be two
integers such that degx (fi ) ≤ dx , degy (fi ) ≤ dy for all i ∈ [1, m]. Let p be the characteristic of
n
Fq , and assume that 2nx +ny dnx x dy y < p. For M an (nx + ny ) × m matrix with entries in Fq , let
(M ) (M )
(f1 , . . . , fnx +ny ) ∈ Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ]nx +ny be defined as
(M )
   
f1 f1
 (M ) 
f2  f2 
   
=M ·
 ..  .
 
 .. 

 . 
  . 
(M ) fm
fnx +ny
(n +n )×m
Then there exists a nonempty open subset O ⊂ Fq x y of the space of (nx + ny ) × m
n +n
matrices such that for any M ∈ O, for any s ∈ [1, nx + ny ], and at any point (x, y) ∈ Fq x y
(M ) (M ) (M ) (M )
such that f1 (x, y) = · · · = fs (x, y) = 0, the derivatives Df1 (x, y), . . . , Dfs (x, y) are
(M ) (M )
linearly independent over Fq . In particular, for any M ∈ O, the sequence (f1 , . . . , fnx +ny ) is
reduced and regular.
5.2. Computing geometric resolutions 69

Proof. This is a reformulation of [132, Thm. A.8.7] in the case of finite fields. In [132, Thm. A.8.7],
this result is stated over the field C, but this statement holds over any field k, provided that an
n nm
extra separability assumption is satisfied. More precisely, set n = nx + ny and let Vs ⊂ k × k
(M ) (M )
be the variety of pairs ((x, y), M ) such that f1 (x, y) = · · · = fs (x, y) = 0. In this setting,
the extra condition that is required for the proposition to hold is that the projection π of Vs
nm
to k must be separable for all s ∈ [1, n] (this is always true in characteristic 0). We refer to
[84, Thm. 4.2] for more details on this separability argument. In our setting, the degree of a
n
generic fiber of π is bounded by 2n dnx x dy y < p using the multi-homogeneous Bézout bound (see
e.g. Proposition 2.48) and hence the separability condition is satisfied.

Since we are looking at polynomial systems over finite fields, we must estimate the size of
the extension of the base field that is required to find with sufficiently large probability a matrix
(M ) (M )
M such that f1 , . . . , fnx +ny is reduced and regular.

Lemma 5.5. Let (f1 , . . . , fm ) ∈ Fq [X1 , . . . , Xnx , Y1 , . . . , Yny ]m be polynomials satisfying the
assumptions of Proposition 5.4 and such that their total degree is bounded above by d ∈ Z≥0 .
Set n = nx + ny and l m
e = (2n + 1) logq (d + 1) + logq (11) .
If M is an n × m matrix with entries in Fqe picked uniformly at random, then the probability
(M ) (M )
that (f1 , . . . , fn ) is a reduced regular sequence is bounded below by 10/11.

Proof. Let Λ denote an n × m matrix with indeterminate entries

λ11 ... λ1m

 
 .. .. .. 
Λ= . . . 
λn1 . . . λnm

and let F1 (Λ, X, Y ), . . . , Fn (Λ, X, Y ) ∈ Fq [X1 , . . . , Xnx , Y1 , . . . , Yny , λ11 , . . . , λnm ] be the polyno-
mials defined as
F1 (Λ, X, Y ) f1 (X, Y )
   
.. ..
=Λ· .
   
 . .
Fn (Λ, X, Y ) fm (X, Y )
For s ∈ [1, n], we consider the s × m matrix Λ(s) obtained by truncating Λ to its s first rows,
a new set of variables {µ1 , . . . , µs−1 } and the following polynomial system:

F1 (Λ(s) , X, Y ) = · · · = Fs (Λ(s) , X, Y ) = 0
 
∂F1 ∂F1 ∂F1 ∂F1
 ∂X ··· ···
i  1 ∂Xnx ∂Y1 ∂Yny 
 h
h  . .. .. .. .. .. 
i
µ1 · · · µs−1 1 ·  ..

. . . . .  = 0 ···
 0
 ∂Fs ∂Fs ∂Fs ∂Fs 

··· ···

∂X1 ∂Xnx ∂Y1 ∂Yny

This is a system of n + s polynomials of degree bounded above by d + 1 in n + s − 1 + ms

variables. By Bézout inequality (see e.g. [72, Thm. 1]), this system defines a variety Vs which
is either empty, or its degree is at most (d + 1)n+s . We remark that if Vs is not empty, then
it has dimension at least ms − 1 since its vanishing ideal is generated by n + s elements. The
sm
Zariski closure of its projection Ws to the space Fq of matrices Λ(s) is either empty, the whole
70 Chapter 5. Asymptotic complexity bounds in arbitrary genus

space or a proper sub-variety. By Proposition 5.4, it must be empty or a proper sub-variety.

Next, we remark that the degree of the image of a variety by a linear projection cannot increase.
Therefore, the sum of the degrees of the irreducible components of Ws is also bounded by
(d + 1)n+s if Ws 6= ∅. In what follows, we let hs (λ11 , . . . , λsm ) denote a polynomial vanishing on
Ws of degree bounded by (d + 1)n+s (we set hs (λ11 , . . . , λsm ) = 1 if Ws = ∅).
The Schwartz-Zippel Lemma implies that the cardinality of the set

···
 
 M11 M1m

 
E =  ... .. ..  ∈ Fnm | h (M , . . . , M ) · · · h (M , . . . , M ) 6= 0


 . .  qe 1 11 1m n 11 nm

Mn1 · · · Mnm
 

is bounded above by q e /11, for the value of e given in the statement.

The proof is concluded by noticing that for any M ∈ E, for any s ∈ [1, n], and for any
n (M ) (M )
(x, y) ∈ Fq such that f1 (x, y) = · · · = fs (x, y) = 0 the derivatives Df1M (x, y), . . . ,
(M ) (M ) (M )
Dfs (x, y) span the normal space at (x, y) to the variety associated with hf1 , . . . , fs i.
(M ) (M )
Hence, f1 , . . . , fn is a reduced regular sequence.

Once we have a reduced regular sequence, we can use Theorem 2.53 to solve the system.
We note that in [25] there is a general assumption that for all s ∈ [1, n] the intermediate ideals
(M ) (M )
hf1 , . . . , fs i define absolutely irreducible varieties. However, the proof of Theorem 2.53
does not require this assumption (this assumption is only required in algorithms for finding a
rational point in [25, Section 6]). To apply the theorem, we need our input to be represented
by division-free straight line programms (DFSLP) as in Definition 2.49 and we can bound the
size of such SLP using Lemma 2.50.
The last ingredient to derive Proposition 5.3 from Theorem 2.53 is an upper bound on
δ = maxi deghf1 , . . . , fi i which was given in Proposition 2.48. Let us now complete the proof of
Proposition 5.3.

5.2.3 Proof of the main complexity result

Proof of Proposition 5.3. Set n = nx + ny . First, we note that if f1 , . . . , fm ∈ Fq [X1 , . . . , Xnx ,
Y1 , . . . , Yny ] is represented by a straight-line program over Fq with space S and time T , then
(M ) (M )
for any e ∈ Z≥0 and any m × n matrix M with entries in Fqe , the sequence f1 , . . . , fn ∈
Fqe [X1 , . . . , Xnx , Y1 , . . . , Yny ] can be represented by a straight-line program over Fqe with space
S 0 and time T 0 , where S 0 = O(S) and T 0 = O(T + m n). We consider the probabilistic Turing
machine which performs the following steps:

1. It chooses an m × n matrix uniformly at random with entries in Fqe , with

l m l m
e = max (2n + 1) logq (d + 1) + logq (11) , logq (60 n4 d δ) ,

n
where d = dx + dy = (`C + 1) h(g), n = nx + ny = Cg + h(g), δ = 2n dnx x dy y =
2
(2h(g))Cg+h(g) `C g . Using the inequalities nx < C g, ny < h(g), dx < h(g)`C , dy < h(g),
we get that e = Og (logq `);
(M ) (M )
2. It constructs the straight-line program representing f1 , . . . , fn with space S 0 = O(S)
and time T 0 = O(T + m n);
5.3. Computing generic `-torsion points 71

3. It applies the probabilistic Turing machine from Theorem 2.53 to compute a geometric
(M ) (M )
resolution of the algebraic set defined by f1 (X) = · · · = fn (X) = 0; By Theorem 2.53,
it returns a geometric resolution ((`1 , . . . , `n ), q(T ), (q1 (T ), . . . , qn (T ))) with probability
(M ) (M )
11/12, provided that f1 (X), . . . , fn (X) is a reduced regular sequence;

4. It computes λ(T ) = GCD(q(T ), f1 (q1 (T ), . . . , qn (T )), . . . , fm (q1 (T ), . . . , qn (T )));

5. It computes ν1 (T ) = q1 (T ) mod λ(T ), . . . , νn (T ) = qn (T ) mod λ(T ) and returns the geo-

metric resolution ((`1 , . . . , `n ), λ(T ), (ν1 (T ), . . . , νn (T ))).

We start by showing that the output of this algorithm is indeed a geometric resolution of
n
the algebraic set V = {x ∈ Fq | f1 (x) = · · · = fm (x) = 0}, assuming that the probabilistic
(M ) (M )
algorithm in Step 3 returns the correct result and that (f1 , . . . , fn ) is a reduced regular
n (M ) (M )
sequence. Let W be the algebraic set {x ∈ Fq | f1 (x) = · · · = fm (x) = 0}. Since
(M ) (M )
hf1 , . . . , fn i ⊂ hf1 , . . . , fm i, we have V ⊂ W . By construction, the algebraic set defined
by the geometric resolution ((`1 , . . . , `n ), λ(T ), (ν1 (T ), . . . , νn (T ))) is precisely the subset of W
where all polynomials f1 , . . . , fm simultaneously vanish.
It remains to prove that this Turing machine runs within the desired complexity. Steps 1
and 2 require negligible time. Step 3 is done within space O((S 0 + n + d)δ 2 log(q e δ)) and
time O((nT
e 0 + n5 )δ(dδ + log(q e δ)) log(q e δ)) (Theorem 2.53), provided that δ is an upper bound

on the degrees of the intermediate ideals. Step 4 is done within space and time bounded by
O(δ
e e log q(T + m)) by evaluating the SLP modulo q(T ) (whose degree is bounded by δ) and
then by computing m GCD using a quasi-linear algorithm. Finally, Step 5 can be done within
time and space O(δ e e log q).
Then, Proposition 2.48 shows
that δ is an upper bound on the degrees of the intermediate
ideals. Using the facts that nxd+d nx = O (`C 2 g ) and ny +dy ≤ (n +d )ny = O (1),


x
x
≤ (n x +d x ) g dy y y g
Lemma 2.50 provides bounds on S and T . Summing these complexities leads to the claimed
complexity estimate. Finally, the probability of success is bounded below by the probability that
(M ) (M )
the sequence f1 , . . . , fn is reduced and regular (Lemma 5.5) multiplied by the probability
of success of the probabilistic Turing machine in Theorem 2.53, namely 10/11 · 11/12 = 5/6.

5.3 Computing generic `-torsion points

Let C be a hyperelliptic curve of genus g over Fq with Weierstrass form Y 2 = f (X) (f monic,
squarefree, and deg(f ) = 2g + 1) and let J be its Jacobian. Let ` > g be a prime not dividing
q. In this section, we define a notion of genericity for `-torsion elements in J and we show
that a geometric resolution for the variety they form can be computed efficiently using the tools
described in Section 5.2. Our starting point is the modelling of the `-torsion sketched by Cantor
in the point (5) of Section 9 of [28]. This section and the next one that deals with the non-generic
cases rely heavily on the Mumford representation detailed in Theorem 1.29.
In what follows, we often also call Mumford representation a pair of polynomials where u is
not monic. In that case, unicity of the representation is no longer guaranteed, but there is no
ambiguity on the element of J represented this way.
In genus 1, the `-torsion points are the points whose abscissae are the roots of the `-division
polynomial, which has degree O(`2 ). For higher genera, Cantor [28] described analogous poly-
nomials δ` and ε` . By Theorem 4.1, for (x, y) the generic point of the curve and ` > g, we
72 Chapter 5. Asymptotic complexity bounds in arbitrary genus

have
x−X x−X
    
` ((x, y) − ∞) = δ` , ε` .
4y 2 4y 2
Let us now restate Theorem 4.13, which is proven in Section 4.2 of Chapter 4 :
The polynomial δ` (X) has degree g and its coefficients are polynomials in Fq [x] of degree
bounded by 31 g`3 +Og (`2 ). The polynomial ε` (X)/y has degree less than g and its coefficients are
rational fractions in Fq (x). The degrees of the numerators and denominators of these coefficients
are bounded by 23 g`3 + Og (`2 ). Furthermore, any root of a denominator is also a root of the
leading coefficient of δ` (X).
Remark that this result is also proven for any non-prime ` > g, it will be used in Section 5.4
where we handle non-generic situations. However, we will also need to define analogues of these
polynomials to describe `P when P is not generic. This is done in Definition 5.9 and we also
remark later on that the previous degree bounds still apply to non generic division polynomials.
Later on, we will need explicit names for these coefficients of δ` and ε` , so we define the
univariate polynomials di and ei (the notation does not show the dependence on ` for simplicity)
such that, after clearing denominators we have:
g g−1
x−X x−X ei (x) i
  X   X
i
δ` = di (x)X , and ε` =y X.
4y 2 i=0
4y 2 i=0
eg (x)

Definition 5.6. In what follows, we shall say that an element of J is `-generic if it has weight
g and the corresponding reduced divisor gi=1 (Pi − ∞) satisfies the following two properties:
P

• For any i, the u-coordinate of the divisor `(Pi − ∞) in Mumford form has degree g;

• For any i 6= j, the u-coordinates of the divisors `(Pi − ∞) and `(Pj − ∞) are coprime.

This implies that the Pi are distinct, and that if an affine point P occurs in the support of a
`(Pi − ∞) then neither P nor −P appears in the support of another `(Pj − ∞).

Proposition 5.7. For any ε > 0, there is a constant D such that for all prime ` > g co-
prime to the base field characteristic, there is a Monte Carlo algorithm which computes an Fqe -
geometric resolution of the sub-variety of J[`] consisting of `-generic `-torsion elements, where
e = Og (log `). The time and space complexities of this algorithm are bounded by Og (`Dg (log q)2+ε )
and it returns the correct result with probability at least 5/6.

Proof. Let D = gi=1 (Pi −∞) be an `-generic divisor in J. We shall consider a system equivalent
P

to `D = 0 but let us first introduce some notation. For each point Pi = (xi , yi ) in the support of
D, we denote by hui , vi i the Mumford form of `(Pi −∞) and by (αij , βij )1≤j≤g the coordinates of
the g points in its support counted with multiplicities, which means that for any i the g roots of
ui are exactly
 the αij , and that for 
any j, 
βij = vi (αij ). Note that using the previous notation,
xi −X xi −X
ui (X) = δ` 4yi2
and vi (X) = ε` 4yi2
.

We have `D = 0 if and only if the sum of the divisors gi=1 `(Pi − ∞) is a principal divisor.
P

The only pole is at infinity, so this is equivalent to the existence of a non-zero function ϕ ∈ Fq (C)
of the form P (X) + Y Q(X) with P and Q two polynomials such that the g 2 points (αij , βij ) are
the zeros of ϕ, with multiplicities. Since we want ϕ to have g 2 affine points of intersection with
the curve C (once again, counted with multiplicities), the polynomial ResY (Y 2 − f, P + Y Q) =
P 2 − f Q2 must have degree g 2 which yields 2 deg(P ) ≤ g 2 and 2 deg(Q) ≤ g 2 − 2g − 1. Exactly
5.3. Computing generic `-torsion points 73

one of those two bounds is even (it depends on the parity of g), and for this particular bound, the
inequality must be an equality, otherwise the degree of the resultant would not be g 2 . Since the
function ϕ is defined up to a multiplicative constant, we can normalize it so that the polynomial
P 2 + f Q2 is monic, which is equivalent to enforce that either P or Q is monic depending on the
parity of g.
For a fixed i ∈ [1, g], requiring the (αij , βij ) to be zeros of ϕ amounts to asking for the αij
to be roots of P (X) + Q(X)vi (X), with multiplicities. Since the αij are by definition the roots
of the ui , `D = 0 is equivalent to g congruence relations P + Qvi ≡ 0 mod ui which we can
rephrase using Cantor’s polynomials:
! !
xi − X xi − X
P (X) + ε` Q(X) ≡ 0 mod δ` . (5.1)
4yi2 4yi2

Thus, for any `-generic divisor, `D = 0 is equivalent to the existence of P and Q satisfying the
above g congruence relations.
The variables are the coefficients of P and Q, as well as the xi and yi . With the degree
conditions and the normalization, we have g 2 − g variables coming from P and Q. Adding
the 2g variables xi and yi , we get a total of g 2 + g variables. Each one of the g congruence
relations (5.1) amounts to g equations providing a total of g 2 conditions on the coefficients of
P and Q. The fact that the (xi , yi ) are points of the curve yields the g additional equations
yi2 = f (xi ). Finally, we have to enforce the `-genericity of the solutions, which can be done by
requiring that i dg (xi ) i<j Res(ui , uj ) 6= 0. Therefore, we get a polynomial system with g 2 + g
Q Q

equations in g 2 + g variables, together with an inequality. We remark that in principle, the

denominators eg (xi ) involved in ε` would generate additional conditions, but by Theorem 4.13
this is already covered by the condition dg (xi ) 6= 0.
In order to apply Proposition 5.3, we now estimate the degrees to which the variables occur
in the equations. We start with the equations coming from (5.1). Each congruence relation is
obtained by reducing P + Qvi , which is a polynomial of degree O(g 2 ) in X, by ui which is of
degree g. We can do it by repeatedly replacing X g by − j<g (dj (xi )/dg (xi ))X j , which we will
P

have to do at most O(g 2 ) times. Since by Theorem 4.13 the dj have degree in Og (`3 ) in xi the
fully reduced polynomial will have coefficients that are fractions for which the degrees of the
numerators and of the denominators are at most Og (`3 ) in the xi variables. In these equations,
the degree in the yi variables and in the variables for the coefficients of P and Q is 1. The
degrees in xi and yi in the curve equations are 2g + 1 and 2 respectively.
It remains to study the degree of the inequality. Each resultant is the determinant of a
2g × 2g Sylvester matrix whose coefficients are the di , which have degrees bounded by Og (`3 ).
Since for any i there are exactly g resultants involving xi in the product, the degree of this
inequality in any xi is in Og (`3 ), and it does not involve the other variables. In order to be able
to use Proposition 5.3, we must model this inequality by an equation, which is done classically
by introducing a new variable T and by using the equation T · i dg (xi ) i<j Res(ui , uj ) = 1.
Q Q

To conclude, we have a polynomial system with two blocks of variables: the 2g variables xi
and yi and the g 2 − g variables coming from the coefficients of P and Q. The degree of the
equations in the first block of variables grows cubically in `, while the degree in the other block
of variables depends only on g. The system therefore verifies the conditions of Proposition 5.3
and the complexity follows, provided that we can show that the system is 0-dimensional and
radical.
Let us consider the sub-variety S ⊂ J[`] consisting of `-generic `-torsion elements, and I
the corresponding ideal. More precisely, we see I as the ideal of a sub-scheme of the `-torsion
74 Chapter 5. Asymptotic complexity bounds in arbitrary genus

scheme, which is the kernel of a finite and étale map because ` is coprime to the characteristic.
Therefore I is 0-dimensional and radical. Since all the elements in S have the same weight g
we can use the Mumford coordinates hu(X), v(X)i with deg u = g and deg v < g − 1 as a
local system of coordinates to represent them. But the polynomial system that we have built
is with the (xi , yi ) coordinates, that is, it generates the ideal I unsym obtained by adjoining to
the equations defining I the 2g equations coming from u(X) = (X − xi ) and yi = v(xi ).
Q

Then we have deg I unsym = g! deg I. By the `-genericity condition, all the fibers in the variety
have exactly g! distinct points corresponding to permuting the (xi , yi ) which are all distinct.
Therefore the radicality of I implies the radicality of I unsym and we can apply Proposition 5.3
to our polynomial system.

We emphasize that, although the algorithm in Proposition 5.7 is Monte Carlo, we expect
that it returns a correct and verifiable result in most of the cases. Indeed, if all the `2g − 1
nonzero `-torsion elements are `-generic (which is the situation that we expect to happen in
most of the cases) and if the algorithm returns the correct result, then we can check that these
elements are indeed `-torsion elements, and that we have all of them. In that favorable case,
the proof of Proposition 5.2 is completed.

5.4 Non-generic cases

For most of the curves, we expect that for all the primes ` considered in Algorithm 6 the set
J[`] contains only `-generic elements (apart from 0), so that the result of the previous section
is sufficient. If this is not the case, then it is very likely that the orbit under the Frobenius
endomorphism of the `-torsion elements computed contains an F` -basis of J[`], so that we can
easily recover the missing elements using the group law or the Frobenius. Still, unless we
could prove otherwise, we can not exclude the case where the set of `-generic `-torsion elements
generate a proper subgroup of J[`] which is stable under the action of ϕ. In that unlikely case,
we would maybe not be able to deduce χ` . An option is then to skip this unlucky ` and proceed
with the algorithm; this would only marginally increase the largest considered `. But then, we
would be left to prove that the number of unlucky `’s is small enough, which seems as hard.
Our only remaining option is to perform a tedious, systematic study of all the non-generic
cases and to show that they can all be modelled by polynomial systems that can be solved within
the target complexity. The number of these systems must also be bounded independently of `,
so that with our setting where g is fixed and q grows to infinity the global complexity remains
the same. All this is the purpose of Subsection 5.4.2. As a warm-up, we will first describe
some simple degeneracy cases and, informally, how to deal with them. Since several causes of
non-genericity may simultaneously appear, we then describe a data structure to encode all the
possible non-generic cases. Then, we detail how to build a polynomial system modelling each of
these cases. Note that all these systems will have more equations (O(g 4 ), see Table 5.2) than
variables (O(g 2 ), see Table 5.1), which is no wonder since we expect them to have no solution
in general.
Lastly, we point out that Subsections 5.4.1 and 5.4.2 can easily be skipped at first reading
as it is only devoted to proving the main theorem of the chapter and will not be used in other
chapters or sections of this thesis.
5.4. Non-generic cases 75

5.4.1 Simple degeneracies

Case 1: Low weight `-torsion elements. In order to compute the `-torsion elements that
satisfy all the conditions of `-genericity except that their weight is less than g, we can proceed as
in the proof of Proposition 5.7 with the following modifications. This time, D = w i=1 (Pi − ∞),
P

and the only difference is that there are w points instead of g. Following the same method, we
search ϕ of the form P (X) + Y Q(X) such that the points in the reduced divisor `(Pi − ∞) are
exactly the zeros of ϕ. We now want ϕ to have gw points of intersection with C instead of g 2 , and
we similarly deduce 2 deg(P ) ≤ gw and 2 deg(Q) ≤ gw − 2g − 1. By similar parity considerations
we deduce that exactly one of these bounds is even, and the corresponding polynomial will be
made monic to normalize the function. The number of variables from P and Q is thus gw − g,
and after adding the 2w variables xi and yi , we have a total of (g + 1)w + w − g variables.
As for the number of equations, the number of congruence relations is now w but the relations
themselves remain unchanged, and we get a total of (g + 1)w equations after adding the w
equations yi2 = f (xi ). Since we keep the degrees unchanged but reduce the number of variables,
the complexity bounds are still valid in this case.

Case 2: Multiple points in the `-torsion divisor. It may happen that the reduced forms
of `-torsion divisors contain multiple points. In that case, the u-coordinate in the Mumford
representation of such a point is not squarefree. Although the modelling by the polynomial
system described in Section 5.3 is still faithful, such multiple points will induce multiplicities since
what we actually compute is the variety describing the points in the reduced divisor. Therefore,
the ideal generated by the polynomial system is not radical in this case. We use the following
workaround: For λ = (λ1 , . . . , λk ) a partition of w, we write a polynomial system generating a
radical ideal whose solutions represent the reduced divisors of the form D = λ1 P1 + · · · + λk Pk −
w ∞. To build this polynomial system, we do as if we were looking for elements of weight k, but
instead of multiplying Pi by `, we multiply it by λi `, using Cantor’s polynomials δλi ` and ελi ` .
This system has the same number of variables and equations as if we were looking for elements
of weight k. Since λi is bounded above by g, the degrees of the equations are multiplied by a
quantity which depends only on g but not on `. Consequently, the complexity bounds are still
valid in this case. To avoid multiplicity problems that could arise from subpartitions of λ, we
add the inequalities xi 6= xj for i 6= j, where xi is the x-coordinate of Pi . Again, this does not
change our complexity estimate.

Case 3: Low weight after multiplication by `. We study here the case where the `-
genericity property that is not verified is that the `(Pi − ∞) are of weight g, all the others being
satisfied. We denote by wi ≤ g the weight of `(Pi − ∞). Then each ui will have degree wi , so
that each congruence relation (5.1) yields only wi equations instead of g. In Cantor’s article (on
top of page 141 in [28]), it is stated that ` · (Pi − ∞) is of weight wi if and only if for any k
such that wi < k ≤ g we have ψ`−k+wi +1 (xi ) = 0 and ψ`−g+wi (xi ) 6= 0, where the polynomials
ψi are efficiently computable and of degrees bounded by Og (`2 ). Therefore the total number
P
of equations is unchanged. Since the function ϕ will have to vanish at i wi points instead of
g 2 , we also reduce the degree of P and Q accordingly. The number of variables from P and Q
thus becomes i wi − g which is smaller than in the generic case, while the number of equations
P

remains the same, and their degrees are also smaller. Thus we can still describe this non-generic
situation with systems that can be handled within the same complexity bounds.
76 Chapter 5. Asymptotic complexity bounds in arbitrary genus

Case 4: Non semi-reduced principal divisor. We now consider the case where the `-
genericity property fails due to the presence of a point of abscissa ξ which appears with positive
multiplicity νi in an `(Pi − ∞) and with a negative multiplicity −νj in another `(Pj − ∞). Let
ν = min(νi , νj ). This event implies that (X − ξ)ν divides both P and Q so that we can write
ϕ(X, Y ) = (X − ξ)ν (Pe (X) + Y Q(X)),
e with Pe coprime to Q.e The number of variables coming
from ϕ is reduced compared to the generic case: we add one (the variable ξ), but the number of
coefficients in Pe is reduced by ν compared to P , and the same is true for Q
e and Q. To write the
conditions on ϕ, we write the congruences exactly like in the generic case and we add conditions
to ensure that the multiplicities are respected. Namely, ui , uj and vi + vj must all be divisible
by (X − ξ)ν , which adds 3ν ≤ 3g equations. The degree in ξ in these equations is bounded by
g 2 . Since this does not depend on `, the complexity result is maintained. The general study
will cover the case where there are several ξ’s at which the semi-reduction genericity assumption
fails. Also, there is no reason why such a root ξ should occur in only two of the `(Pi − ∞)’s.
Such a situation will be also taken into account in Section 5.4.2.

Case 5: Multiplicity in `D. The last situation that could lead to not satisfying `-genericity
is when the same point is shared within different `(Pi − ∞), which causes some trouble as the
congruence relations of the generic case will not be able to handle the subsequent multiplicity.
Note that if the multiplicity occurs only within a single `(Pi − ∞) this is already dealt within the
generic case. One can view our method as using the Chinese remainder theorem on the modular
conditions (5.1) to see that multiplicities within a single congruence is handled whereas common
factors within different ui -polynomials are an obstacle that needs special strategies. There is
some similarities with the previous case that also implies a common factor between two different
ui ’s.
We devise the following workaround: instead of considering the congruences modulo the
ui ’s separately, we group them into a single congruence of the form P + QV ≡ 0 mod U , with
U = i ui and V a polynomial whose coefficients shall be new variables such that V ≡ vi mod ui
Q

for all i. Note that if some non semi-reduced case occurs simultaneously, U must actually be
divided by the aforementioned X − ξ; such situations will be dealt with later, in the general
study (Section 5.4.2). In order for V to encode enough information and ensure that the condition
P + QV ≡ 0 mod U enforces a function with exactly the correct principal divisor, we have to
follow Mumford’s representation and add the condition U |V 2 −f , with deg V < deg U . Together
with the other conditions on U and V , we then have existence and unicity (up to a constant
factor): they are the result of Cantor’s composition algorithm.
In order to write the polynomial system modelling this situation, some care must be taken
so as to stay within the scope of Proposition 5.3. The polynomial U is of degree g 2 and its
coefficients are polynomials in the xi ’s of degrees bounded by Og (`3 ). New variables are added
for the coordinates of V . For each i, the condition V ≡ vi mod ui is converted in O(g) equations,
with degrees Og (`3 ) in xi and 1 in the coordinates of V . The condition U |V 2 − f contributes
to O(g 2 ) additional equations, each of them of degree 2 in the coordinates of V , and degree
Og (`3 ) in the coordinates xi . And finally, the equation P + QV ≡ 0 mod U , contributes also
to O(g 2 ) equations, each of them of degree 1 in the coordinates of V , P and Q, and of degree
Og (`3 ) in the coordinates xi . Skipping the details, we can again apply Proposition 5.3 and get
the expected complexity.
5.4. Non-generic cases 77

5.4.2 Combining all possible degeneracies

A data structure to describe each type of non-genericity. We want to describe a family
of polynomial systems that covers all the possible non-generic cases, possibly mixing all kind of
problems that have been listed. We begin by grouping together non-genericity situations that
can be covered by the same polynomial system.
We consider an `-torsion divisor D of weight w ≤ g (like in case 1). Next, a partition λ =
(λ1 , . . . , λk ) of w is picked to represent the multiplicity pattern in the u-coordinate of the `-torsion
divisor, as in case 2 so that D = ki=1 λi (Pi − ∞). Then, a vector t = (t1 , . . . , tk ) is chosen, to
P

represent the weights of the Pi after multiplication by λi ` as in case 3: For i in [1, k], the reduced
divisor λi `(Pi − ∞) is of weight ti . Then, we need to consider how many common or opposite
points these divisors are in their support to take into account the cases 4 and 5. We denote by
Q1 , . . . , Qs the points in the union of the supports of all the reduced divisors λi `(Pi −∞), keeping
only one point in each orbit under the hyperelliptic involution. We represent the non-genericity
by a k × s matrix M such that its non-zero entries mij satisfy mij = ordQj (λi `(Pi − ∞)) when
Qj is in the support of λi `(Pi − ∞) or mij = − ordQ0j (λi `(Pi − ∞)) when the hyperelliptic
conjugate Q0j of Qj is in the support. Note that this matrix, that we shall call the matrix of
shared points, represents both multiplicities and non-semi-reduction. Since the row i represents
what happens with points in the support of λi `(Pi − ∞), which is of weight ti , the sum of the
absolute values of the entries of the row i of M is equal to ti .
Also, by construction, in each column, there is at least one non-zero entry. An additional
complication arises when one of the Pi is a ramification point, i.e. when its y-coordinate is
zero, because this would cause multiplicities if care is not taken, leading to non-radicality of the
polynomial system we build. Since this corresponds to Pi − ∞ being of order 2, the weight ti is
equal to λi ` mod 2, namely 0 or 1. If ti = 0, then the divisor D − λi (Pi − ∞) is also an `-torsion
divisor of weight w − λi , so that we can reconstruct D from another polynomial system. There
is however no obvious way to preclude the possibility ti = 1. Therefore, we will encode the fact
that Pi is a ramification point by a bit i that can be set only in the cases where ti = 1 and
λi = 1.
A tuple (w, λ = (λ1 , . . . , λk ), t = (t1 , . . . , tk ),  = (1 , . . . , k ), M ) is from now on the piece of
data with which we represent a non-generic situation, and a polynomial system will be associated
to each tuple. Changing the order of the columns of M amounts to permuting the points Qj .
Also, changing the sign of all the entries of a column j corresponds to taking the opposite of the
point Qj . While it would not change the final complexity not to do so, it therefore makes sense to
consider only normalized tuples, in the sense that the columns of M are sorted in lexicographical
order, and the choice between a point Qj and its opposite is done so that the sum of all elements
in the corresponding column is nonnegative. We remark that this is not enough to guarantee
that two normalized tuples do not describe similar situations. For instance, if λ = (1, . . . , 1)
and two ti values are equal, then permuting the two corresponding rows could lead to another
normalized matrix that would describe the same situation. This is not a problem for the general
algorithm: we might get the same `-torsion elements from two different systems, but what is
important to us is non-multiplicity (i.e. radicality of the ideal) in each individual system.

Definition 5.8. A normalized non-genericity tuple is a tuple (w, λ, t, , M ), where 1 ≤ w ≤ g

is an integer, λ = (λ1 , . . . , λk ) is a partition of w, t and  are vectors t = (t1 , . . . , tk ) and
 = (1 , . . . , k ) of the same length as λ with 1 ≤ ti ≤ g and i ∈ {0, 1}, where i can be 1 only
if ti = 1 and λi = 1, and finally M is a matrix with k rows and s columns, where 0 ≤ s ≤ g k,
and its entries are integers such that:
78 Chapter 5. Asymptotic complexity bounds in arbitrary genus

• For all 1 ≤ i ≤ k, the sum of the absolute values of the entries on the row i is equal to ti ;

• The columns are sorted in lexicographical order;

• The sum of the rows of the matrix is a vector whose coordinates are nonnegative.
From the discussion above, any `-torsion element is described by (at least) one normalized
non-genericity tuple. In the following we will give a polynomial system for each normalized
non-genericity tuple, so that all `-torsion elements described by it are modelled by this system.
Furthermore, the system will have the properties required to apply Proposition 5.3, so that the
complexity result will follow.
Before starting this, we discuss briefly a bound on the number of normalized non-genericity
tuples. Assuming everything is always of maximal size, and not sorted, we have g choices for
2
w, then at most g g choices for λ and t, at most 2g choices for , and finally at most (g 2g+1 )g
3
choices for M , which gives g O(g ) . As bad as
 it is, such a factor that depends only on g will not
hinder the final complexity estimate in Og (log q)O(g) , as explained in Section 5.1.

Non-generic division polynomials. The expression of λi ` · (Pi − ∞) in Mumford represen-

tation will be the same as in the generic case when its weight ti is equal to g and Theorem 4.13
can be applied. But when ti is strictly less than g, the weight-g coordinate system is no longer
available; this is explicitly visible by the fact that the denominator eg (xi ) of the coefficients of
the v-polynomial vanishes.
Therefore we need to use a weight-t coordinate system for describing a non-generic divisor
λi ` · (Pi − ∞) in Mumford representation. In this paragraph, in order to keep simple notation,
we will work with `(Pi − ∞), keeping in mind that we do not impose any condition on `, so that
we can later replace ` by λi `.
We consider, for 1 ≤ t < g, the set V`,t of points of the curve which are mapped to a weight-t
divisor after multiplication by `:

V`,t = {(x, y) ∈ C | ` · ((x, y) − ∞) is of weight t} .

This is a (possibly empty) variety of dimension 0 that can be described with the classical (generic)
division polynomials of Cantor: we define

∆`,t = GCD(ψ` (x), ψ`−1 (x), . . . , +ψ`−g+t+1 (x)),

so that V`,t is precisely the set of points (x, y) for which ∆`,t (x) = 0 and ψ`−g+t (x) 6= 0, as stated
by Cantor in [28] on page 141. The polynomial ψ` is essentially the square root of the leading
coefficient of δ` . It can be computed efficiently and has degree in Og (`2 ) by Theorem 8.17 of [28].
To avoid multiplicities, we define ∆ ˜ `,t (x) the square-free polynomial whose roots are exactly the
roots of ∆`,t (x) that are not roots of ψ`−g+t (x). The degree of ∆ ˜ `,t (x) is again bounded by
2
Og (` ). Furthermore since the points of V`,t come in pairs of conjugate points sharing the same
x-value, the degree of V`,t is 2 deg ∆ ˜ `,t (x).

Definition 5.9. The non-generic division polynomials u`,t and v`,t are the polynomials in X
˜ `,t (x), y 2 − f (x)) such that
with coefficients in Fp [x, y]/(∆
D E
` · ((x, y) − ∞) = u`,t (X), v`,t (X) ,

in weight-t Mumford representation: u`,t (X) is monic of degree t, v`,t (X) is of degree at most
t − 1 and they satisfy u`,t | v2`,t − f .
5.4. Non-generic cases 79

Just like for the classical division polynomials, the coefficients of u`,t (X) and of y1 v`,t (X)
˜ `,t (x) (they do not depend on y) and we can choose representatives of them that
are in Fp [x]/∆
are polynomials of degree less than deg ∆ ˜ `,t (x). Hence, the bounds given in Theorem 4.13 are
also valid for the non-generic division polynomials; and since there are no denominators in the
coefficients of v`,t (X), the other part of Theorem 4.13 also holds trivially.
The non-generic division polynomials can be computed efficiently, once the classical division
polynomials are known: the polynomial ∆ ˜ `,t (x) can be easily deduced, and then working in the
quotient algebra yields the result in a time O e g (`2 ), which is negligible compared to the other
parts of the algorithm.

5.4.3 Polynomial system derived from a normalized non-genericity tuple.

We now want to write a polynomial system whose solutions are the `-torsion elements following
a given normalized non-genericity tuple (w, λ, t, , M ).
First, we need variables for the coordinates of the Pi such that the `-torsion element is
D = ki=1 λi (Pi − ∞), with Pi 6= ±Pj for all i 6= j. As a consequence, we introduce 2k variables
P

for the coordinates (xi , yi ) of all the points Pi . Since these points are on the curve, they satisfy
yi2 = f (xi ), however if Pi is a ramification point this can be simplified into yi = 0 = f (xi ), which
avoids the multiplicities. We get a first set of equations
(
yi2 = f (xi ) 6= 0, for all i in [1, k] such that i = 0,
(Sys.1)
yi = f (xi ) = 0, for all i in [1, k] such that i = 1.

As we just discussed, we must model the fact that Pi 6= ±Pj for i 6= j. This is done via the
following set of inequalities:

xi 6= xj , for all i, j in [1, k] such that i 6= j. (Sys.2)

The next step is to enforce the fact that the element λi `(Pi − ∞) is of weight ti . For the
indices for which ti < g, this is encoded by the equation defining Vλi `,ti :
(
˜ λ `,t (xi ) = 0,
∆ i i
for all i in [1, k] such that ti < g, (Sys.3)
dti (xi ) 6= 0,

while for the indices for which ti = g, this is encoded by the non-vanishing of the leading
coefficient of the Cantor polynomial in degree λi `:

dg (xi ) 6= 0, for all i in [1, k] such that ti = g. (Sys.4)

We now need to model the fact that the λi `(Pi − ∞) satisfy the conditions given by the
matrix M . We write λi `(Pi − ∞) = hui (X), vi (X)i in Mumford representation, where ui (X)
and vi (X) are Cantor’s classical division polynomials in degree λi ` if ti = g or the non-generic
division polynomials uλi `,ti and vλi `,ti , if ti < g. In both cases, these are polynomials in X
whose coefficients are polynomials in xi and yi . Recall that the entries of M , denoted by
(mij )i∈[1,k],j∈[1,s] , are such that mij is the order of Qj in λi `(Pi − ∞) if it is positive, or the
opposite of the order of Q0j if it is negative. To this effect, we introduce s new variables ξj for
80 Chapter 5. Asymptotic complexity bounds in arbitrary genus

the abscissae of the Qj , and the following equations enforce the multiplicities:

(n)
ui (ξj ) = 0, for all i, j in [1, k] × [1, s] and for all n ≤ |mij | − 1 (Sys.5)
(|m |)
ui ij (ξj ) 6= 0, for all i, j in [1, k] × [1, s] (Sys.6)
0
vi (ξj ) − vi0 (ξj ) = 0, for all i, i , j such that mij mi0 j > 0 (Sys.7)
0
vi (ξj ) + vi0 (ξj ) = 0, for all i, i , j such that mij mi0 j < 0 (Sys.8)
0
ξj 6= ξj 0 , for all j 6= j . (Sys.9)

(n)
In Equations Sys.5 and Sys.6, the notation ui is for the n-th derivative of ui . This simple way
of describing multiple roots is valid because the characteristic is large enough.
The next step of the construction is to consider a semi-reduced version of the divisor `D =
Pk
i=1 λi `(Pi − ∞). This semi-reduction process can be described directly on the matrix M : if
two entries in a same column have opposite signs, a semi-reduction can occur (corresponding to
subtracting the principal divisor of the function (x − ξj )), thus reducing the difference between
these entries. This semi-reduction can continue until one of these two entries reaches zero. This
whole process can be repeated as long as there are still columns containing entries with opposite
signs. This is formalized in Algorithm 7, which takes as input a matrix M and returns a matrix
M
f with the same dimensions such that if M describes all the multiplicities in a divisor, then M f
describes all the multiplicities of a semi-reduced divisor equivalent to the input divisor. More
precisely, the matrix Mf satisfies the following properties: (1) In each column, all elements are
nonnegative; (2) The sum of the rows of M equals the sum of the rows of M f; (3) For all i, j
such that Mi,j is nonnegative, Mij ≤ Mij .
f

Data: M the k × s matrix of shared points of the system

Result: M f, the matrix after semi-reduction
f ← k × s zero matrix
M
for j from 1 to s do
µj ← ki=1 Mij
P

for i from 1 to k do
if Mij > 0 then
fij ← min(Mij , µj )
M
µj ← µj − M
fij
else
fij ← 0
M
end
end
end
return M f
Algorithm 7: Reducing the matrix of shared points

The function ϕ that we will use to model the principality of the divisor `D will have two
parts: a product of “vertical lines” corresponding to semi-reductions, and a part of the form
P (X)+Y Q(X), where P and Q are coprime. Modelling the existence of this second part requires
to introduce new entities u
ei that are the ui polynomials from which we remove the linear factors
coming from semi-reduction as described by M f. Formally, we have the following equations,
5.4. Non-generic cases 81

defining u
ei :
s
(X − ξj )|mij |−m
Y
ui (X) = u
ei (X) e ij , for all i ∈ [1, k]. (Sys.10)
j=1

Indeed, by definition of the matrix M , the factor (X − ξj )|mij | divides exactly ui (X), and the
factor (X − ξj )me ij divides exactly u
ei (X). In order to express these conditions efficiently in the
polynomial system, we introduce new variables for the coefficients of the u ei polynomials.
Since we are now dealing with a semi-reduced divisor, we can consider its Mumford repre-
sentation, i.e. two polynomials U and V with the following properties:
k
Y
U= u
ei , U |V 2 − f, (Sys.11)
i=1
V ≡ vi mod u
ei , for all i ∈ [1, k]. (Sys.12)

The expression of U is simple enough, so we do not have to introduce new variables for its
coefficients. However, this will be necessary for the coefficients of the V polynomial. Finally, in
order to impose that the semi-reduced part of ϕ has exactly the zeros described by this divisor,
we have the equation
P + QV ≡ 0 mod U, (Sys.13)
which is expressed with new variables for the coefficients of P and Q.
In Table 5.1, we summarize all the variables used in the polynomial system and count them.
A key quantity for this count is the degree of U which is the sum of the degrees of the u ei ’s. It
can be computed directly from the tuple (w, λ, t, , M ). Then, to ensure existence and unicity of
the V polynomial to represent the semi-reduced divisor, we have to impose that deg V < deg U ,
so that we have exactly deg U variables for the coefficients of V . For the polynomials P and Q,
we need the degree of P 2 − Q2 f to be exactly deg U . After a normalization like in Section 5.3
depending on the parity of deg U , we get deg U − g variables for their coefficients.

Variables Number of variables Bound

Coordinates (xi , yi ) of Pi 2k 2g
Abscissae ξj of shared points s, column-size of the matrix M g2
deg U = i (ti − j (|mij | − m g2
P P
Coefficients of the uei polynomials e ij ))
Coefficients of the V polynomial deg U g2
Coefficients of the P and Q polynomials deg U − g g2 − g
Total s + 2k + 3 deg U − g 4g 2 + g

Table 5.1: Summary of the variables in the polynomial system corresponding to a normalized
non-genericity tuple (w, λ, t, , M ).

In order to apply Proposition 5.3, we need to evaluate the degrees of all the equations and
inequalities that we have listed, with respect to two groups of variables: The first group contains
just the variables xi and yi , and we will denote deg1 (f ) the degree of a polynomial f with respect
to those variables (said otherwise, deg1 (f ) is the degree of f if we consider only the symbols
xi , yi as variables, and all the other indeterminates are considered as parameters). The second
group of variables contains all the other indeterminates and the degree with respect to this group
is denoted by deg2 .
The crucial point is to ensure that each polynomial equation has a deg1 bounded by Og (`3 ),
while deg2 is bounded by Og (1). For the inequalities, we require the same degree conditions:
82 Chapter 5. Asymptotic complexity bounds in arbitrary genus

Indeed, an inequality f 6= 0 can be modeled by the equality T · f − 1 = 0, where T is a

fresh variable that belongs to our second group of variables. This trick requires only one more
variable for each inequality and the degree of the equation T · f − 1 = 0 is only one more than
the degree of the inequality. Since the number of inequalities is bounded by Og (1), the number
of extra variables required in the second group will not impact the asymptotic complexity (the
second group already contains Og (1) variables). We remark that the input of the geometric
resolution algorithm over fields of characteristic 0 in [64] allows inequalities. However, we use
the aforementioned trick to model inequalities by equalities since the solving method that we
use is the variant for positive characteristic whose complexity analysis is given in [25, Thm. 4.8].
The number of equations and inequalities and their degrees with respect to the two groups
of variables can be easily checked and are summarized in Table 5.2.

Equations reference Number of equations (and bound) deg1 deg2

Eq. and Ineq. Sys.1 2k ≤ 2g 2g + 1 0
InEq. Sys.2 k(k − 1)/2 ≤ g(g − 1)/2 1 0
Eq. and Ineq. Sys.3 ≤ 2g Og (`3 ) 0
InEq. Sys.4 ≤g Og (`3 ) 0
Pk Ps 4
Eq. Sys.5 i=1 j=1 |mij | ≤ g Og (`3 ) ≤g
InEq. Sys.6 ks ≤ g 3 Og (`3 ) ≤g
Eq. Sys.7 and Sys.8 ≤ k2 s ≤ g4 Og (`3 ) ≤g
InEq. Sys.9 ≤ s2 ≤ g 4 0 1
Pk 2
Eq. Sys.10 i=1 ti ≤ g Og (`3 ) ≤g
Eq. Sys.11 deg U ≤ g 2 0 O(g 3 )
Pk
Eq. Sys.12 i=1 deg uei ≤ g 2 Og (`3 ) O(g 2 )
Eq. Sys.13 deg U ≤ g 2 0 O(g 3 )

Table 5.2: Summary of the degrees of the equations in the polynomial system corresponding to
a normalized non-genericity tuple (w, λ, t, , M ).

Finally, since we have been very careful in describing elements that are `-torsion points on J,
without room for parasite solutions or multiplicities, we can again appeal to the finite and étale
property of multiplication by ` in J to deduce that the system is 0-dimensional and radical.
Therefore, by Proposition 5.3, each system can be solved in the claimed complexity bound.
To conclude the proof of Proposition 5.2, and hence of our main result, we need a few more
observations.
First, notice that the solutions of our polynomial systems can be grouped by weight of the
`-torsion divisor: once geometric resolutions of two 0-dimensional sets V1 and V2 are known, a
geometric resolution of V1 ∪V2 can be computed very efficiently. The strategy to do so is to change
the primitive element of the geometric resolutions for a random element, so that both resolution
share the same primitive element. This can done within complexity linear in the number of
variables and polynomial in deg(V1 ∪ V2 ) using Algorithm 6 in [64]. Then, computing the LCM
of the univariate polynomials of the geometric resolutions and interpolating the parametrization
provides a geometric resolution of V1 ∪ V2 . Using this procedure for regrouping the solutions
of all the systems derived from the non-degeneracy tuples with the same weight w provides
geometric solutions of Jw [`] within the claimed complexity.
Finally, we need to transform the Monte Carlo algorithm from Proposition 5.3 in a Las Vegas
algorithm. This can be easily achieved since the probability that the Monte Carlo algorithm
succeeds is bounded below by a quantity which does not depend on the input size, and the output
5.4. Non-generic cases 83

can be verified since we know that the sum of the degrees of the varieties Jw [`] for w ∈ [1, g] must
equal `2g −1. Consequently, once all polynomial systems corresponding to non-generic situations
have been solved, it is easy to count the number of `-torsion elements found and to check that
none of them is missing by comparing their number with the theoretical value `2g − 1. The
Las Vegas algorithm consists in repeating the Monte Carlo algorithm until the result is verified
and is correct (i.e. all elements found are `-torsion elements and none of them is missing). The
expected complexity of the Las Vegas variant equals the complexity of the Monte Carlo variant
up to multiplication by a constant. This concludes the proof of Proposition 5.2.
84 Chapter 5. Asymptotic complexity bounds in arbitrary genus
 Chapter 6

The case of genus-3 hyperelliptic

curves with RM

Contrary to p-adic methods that have been adapted to any genus, implementations of `-adic
point-counting algorithms were limited to genus 1 and 2, probably because of the lack of cryp-
tographic applications of genus-3 curves but also because such an algorithm would very likely
have a prohibitive complexity that would impede any practical attempt. In fact, the complexity
of a genus-3 analogue of Schoof’s algorithm is subject to speculations as mentioned in [70] with
12
an estimation in O(log
e q) that is prohibitive indeed. However, as in genus 2, we may try to
find easier instances and in particular consider the RM case.
The aim of this chapter is thus to show — both with theoretical proofs and practical experi-
ments — that the complexity of `-adic methods for genus-3 hyperelliptic curves can be dramati-
cally decreased as soon as an explicitly computable non-integer endomorphism η ∈ End(Jac(C))
is known. More precisely, we consider C a genus-3 hyperelliptic curve with explicit RM by Z[η]
in the sense of Definition 3.3. This means that we have explicit formulas describing η(P − P∞ )
(u)
for P a generic point of C. By explicit formulas, we mean polynomials (ηi (x, y))i∈{0,1,2,3} and
(v)
(ηi (x, y))i∈{0,1,2,3} in Fq [x, y], such that, when C is given in odd-degree Weierstrass form, the
DP E
3 (u) iP 2 (v) (v)
i
Mumford coordinates of η(x, y) are i=0 ηi (x, y)X , i=0 ηi (x, y)/η3 (x, y) X , where
(x, y) is the generic point of the curve. In cases where C does not have an odd-degree Weier-
strass model, we can work in an extension of degree at most 8 of the base field in order to ensure
the existence of a rational Weierstrass point.
Examples of curves with RM are given by modular curves. For instance, the genus-3 curve
y 2 = x7 + 3x6 + 2x5 − x4 − 2x3 − 2x2 − x − 1 is a quotient of X0 (284) and therefore has real
multiplication by an element of Q[x]/(x3 − 3x − 1). This follows from the properties of the Hecke
operators as explained in [130, Chapter 7]. Based on this theory, algorithms for constructing such
curves are explained in [50]; however the explicit expression for the real endomorphism is not
given. We expect that tracking the Hecke correspondences along their construction, and using
techniques like in [144] to reconstruct the rational fractions describing the real endomorphism
could solve this question. In any case, these are only isolated points in the moduli space.
Larger families are obtained from cyclotomic covering. This line of research has produced several
families of hyperelliptic genus-3 curves having explicit RM by Z[2 cos(2π/7)]. In particular, such
explicit families are given in [101] and [138], and explicit formulas for their RM endomorphism
are obtained in [87]. We use the 1-dimensional family of curves from [138, Theorem 1 with
p = 7] for our experiments. Other families of genus-3 curves (but not necessarily hyperelliptic)

85
86 Chapter 6. The case of genus-3 hyperelliptic curves with RM

with RM have been made explicit in [23, Chapter 2], following [43]. We would like to point out
that within the moduli space of complex polarized abelian varieties of dimension 3, those with
RM by a fixed order in a cubic field form a moduli space of codimension 3 [19, Sec. 9.2]. Since
Jacobians of hyperelliptic curves form a codimension 1 space, we would expect the moduli space
of hyperelliptic curves of genus 3 with RM by a given cubic order to have dimension 2.
We insist on the fact that all the O() and the O()e notation used throughout the chapter
should be understood up to a multiplicative constant which may depend on the ring Z[η] and
(u) (v)
on the degrees of the polynomials ηi and ηi . There are natural families of curves for which
these degrees are bounded by an absolute constant and for which Z[η] is fixed: reductions at
primes (of good reduction) of a hyperelliptic curve with explicit RM defined over a number field.
Most of this chapter is joint work with Pierrick Gaudry and Pierre-Jean Spaenlehauer and is to
appear as [2].

Organization of the chapter. In Section 6.1 we give an overview of both our algorithm and
its complexity. The main task is the computation of kernels of some endomorphisms detailed
in Section 6.3. This is achieved by solving a polynomial system using resultants. Section 6.4
is devoted to implementation of the algorithm using Gröbner bases instead of resultants and
ending with an exponential collision search which can be run massively in parallel. Indeed,
although using Gröbner bases seems to be more efficient in practice, we do not see any hope of
proving with rigorous arguments that it is asymptotically competitive.

6.1 Overview of the algorithm

Let C be a genus-3 hyperelliptic curve over a finite field Fq with explicit RM, and let η be the
given explicit endomorphism. We denote by µ0 , µ1 , µ2 the coefficients of the minimal polynomial
T 3 + µ2 T 2 + µ1 T + µ0 of η over Q.

6.1.1 The characteristic equation of the Frobenius

The characteristic polynomial of the Frobenius endomorphism π is of the form χπ (T ) = T 6 −
σ1 T 5 + σ2 T 4 − σ3 T 3 + qσ2 T 2 − q 2 σ1 T + q 3 , and Weil’s bounds give
√
|σ1 | ≤ 6 q, |σ2 | ≤ 15q, |σ3 | ≤ 20q 3/2 .

In order to take advantage of the explicit RM, we consider the endomorphism ψ = π + π ∨ ,

for which we can derive the real Weil polynomial χψ (T ) = T 3 − σ1 T 2 + (σ2 − 3q)T − (σ3 − 2qσ1 ),
which corresponds to the characteristic polynomial of ψ viewed as an element of the real subfield
of End(Jac(C)) ⊗ Q. The endomorphism ψ belongs to the ring of integers of Q(η). The ring
Z[η] might be a proper sub-order of the ring of integers, so let us call ∆ its index, so that ψ can
be written ψ = a + bη + cη 2 , where a, b, c are rationals with a denominator that divides ∆. By
computing formally the characteristic polynomial of a + bη + cη 2 in Q(η) and by equating it with
the expression for the characteristic polynomial of χψ (T ) , we obtain a direct way to compute
σ1 , σ2 and σ3 in terms of a, b, c:
σ1 = 3 a − b µ2 − 2 c µ1 + c µ22 ,
σ2 − 3q = 3 a2 − 2 a b µ2 + 2 a c (µ22 − 2µ1 ) + b2 µ1 + 3 b c µ0 − b c µ1 µ2 −
c2 (2 µ0 µ2 + µ21 ) , (6.1)
σ3 − 2qσ1 = a3 − a2 b µ2 + a2 c (µ22 − 2µ1 ) + a b2 µ1 + a b c (3 µ0 − µ1 µ2 ) +
a c2 (µ21 − 2 µ0 µ2 ) − b3 µ0 + b2 c µ0 µ2 − b c2 µ0 µ1 + c3 µ20 .
6.1. Overview of the algorithm 87

√
In Section 6.2.1, it is shown that the coefficients a, b and c can be bounded in O( q). More
precisely, we denote by Cabc a constant that depends only on η such that their absolute values
√
are bounded by Cabc q. Since these bounds are much smaller than the bounds for σ1 , σ2 , σ3 ,
it makes sense to design an algorithm that reconstruct these coefficients of ψ instead of the
coefficients of χπ as in the classical Schoof algorithm, and this is what we are going to do later
on.
Another important bound that we need concerns the size of small elements that can be
found in ideals of Z[η]. Let ` be a prime that splits completely in Z[η], so that we can write
` = p1 p2 p3 , where the pi ’s are distinct prime ideals of norm `. In Section 6.2.2, it is shown that
each pi contains a non-zero element αi = ai + bi η + ci η 2 , where ai , bi and ci are integers and are
bounded in absolute value by O(`1/3 ).

6.1.2 A point-counting algorithm

Our genus-3 RM point counting algorithm is Algorithm 8. We give a description of it, allowing
some black-box primitives that will be detailed in dedicated sections. As mentioned above, we
will work with the a, b, c coefficients of the ψ endomorphism. More precisely, we compute
their values modulo sufficiently many completely split primes ` until we can deduce their values
from the bounds of Lemma 6.1 from Section 6.2.1 by the Chinese Remainder Theorem, tak-
ing into account their potential denominator ∆. Then the coefficients of χπ are deduced by
Equations (6.1).
We now explain how the algorithm works for a given split `. First its decomposition as a
product of prime ideals ` Z[η] = p1 p2 p3 is computed, and for each prime ideal pi , a non-zero
element αi of pi is found with a small representation αi = ai + bi η + ci η 2 as in Lemma 6.2 of
Section 6.2.2. The kernel of αi is denoted by J[αi ] and it contains a subgroup Gi isomorphic
to Z /` Z × Z /` Z, since the norm of αi is a small multiple of `. We further denote by λi the
eigenvalue of η in J[`] such that pi is the ideal (`, η − λi ).
On Gi ⊂ J[αi ], the endomorphism η acts as the multiplication by λi . Therefore, ψ =
a + bη + cη 2 also acts as a scalar multiplication on this 2-dimensional space, and we write
ki ∈ Z /` Z the corresponding eigenvalue: for any Di in Gi , we have ψ(Di ) = ki Di . On the other
hand, from the definition of ψ, it follows that ψπ = π 2 + q. Therefore, if such a Di is known,
we can test which value of ki ∈ Z /` Z satisfies

ki π(Di ) = π 2 (Di ) + qDi . (6.2)

Since ` is a prime and Di is of order exactly `, this is also the case for π(Di ). Finding ki can then
be seen as a discrete logarithm problem in the subgroup of order ` generated by π(Di ); hence
the solution is unique. Equating the two expressions for ψ, we get explicit relations between a,
b, c modulo `:
a + bλi + cλ2i ≡ ki mod `.
Therefore we have a linear system of three equations in three unknowns, the determinant of
which is the Vandermonde determinant of the λi , which are distinct by hypothesis. Hence the
system can be solved and it has a unique solution modulo `.
It remains to show how to construct a divisor Di in Gi , i.e. an element of order ` in the
kernel J[αi ]. Since an explicit expression of η as an endomorphism of the Jacobian of C is known,
an explicit expression can be deduced for αi , using the explicit group law. The coordinates of
the elements of this kernel are solutions of a polynomial system that can be directly derived
from this expression of αi . Using standard techniques, it is possible to find the solutions of this
88 Chapter 6. The case of genus-3 hyperelliptic curves with RM

input : q an odd prime power, and f ∈ Fq [X] a monic squarefree polynomial of degree
7 such that the curve Y 2 = f (X) has explicit RM by Z[η].
output: The characteristic polynomial χπ ∈ Z[T ] of the Frobenius endomorphism on
the Jacobian J of the curve.
R ← 1;
√
while R ≤ 2 ∆ Cabc q + 1 do
Pick the next prime ` that satisfies conditions (C1) to (C4);
Compute the ideal decomposition ` Z[η] = p1 p2 p3 , corresponding to the eigenvalues
λ1 , λ2 , λ3 of η in J[`] ;
for i ← 1 to 3 do
Compute a small element αi of pi as in Lemma 6.2;
Compute a non-zero element Di of order ` in J[αi ] ;
Find the unique ki ∈ Z /` Z such that ki π(Di ) = π 2 (Di ) + qDi ;
end
Find the unique triple (a, b, c) in (Z /` Z)3 such that a + bλi + cλ2i = ki , for i in
{1, 2, 3} ;
R ← R · `;
end
Reconstruct (a, b, c) using the Chinese Remainder Theorem ;
Deduce χπ from Equations (6.1).
Algorithm 8: Overview of our genus-3 RM point-counting algorithm

system, perhaps in a finite extension of the base field (of degree bounded by the degree of the
ideal generated by the system, i.e. in O(`2 )), from which divisors in J[αi ] can be constructed.
Multiplying by the appropriate cofactor, we can reach all the elements of Gi ; but we stop as
soon as we get a non-trivial one.
We summarize the conditions that must be satisfied by the primes ` that we work with:
(C1) ` must be different from the characteristic of the base field;

(C2) ` must be coprime to the discriminant of the minimal polynomial of η;

(C3) there must exist αi ∈ pi as in Lemma 6.2 with norm non-divisible by `3 for i ∈ {1, 2, 3};

(C4) the ideal ` Z[η] must split completely.

The first 3 conditions eliminate only a finite number of `’s that depends only on η, while the
last one eliminates a constant proportion. The condition (C3) implies that there is a unique
subgroup Gi of order `2 in J[αi ] (our description of the algorithm could actually be adapted to
handle the cases where this is not true).

6.1.3 Complexity overview

The field Q(η) is of degree 3, so its Galois group has order at most 6 and by Chebotarev’s density
theorem the density of primes that split completely is at least 1/6. Therefore the main loop is
done O(log q/ log log q) times, with primes ` that are in O(log q). All the steps that take place
in the number field take a negligible time. For instance, a small generator like in Lemma 6.2
can be found by exhaustive search: only O(`) trials are needed since we are searching over all
elements of the form a + bη + cη 2 , with |a|, |b|, |c| in O(`1/3 ).
6.2. Bounds for Algorithm 8 89

The bottleneck of the algorithm is the computation of a non-zero element of order ` in the
kernel J[αi ] of αi . This part will be treated in detail in Section 6.3, where it is shown to be
e 4 ) operations in Fq . The output is a divisor Di of order ` in J[αi ] that is defined
feasible in O(`
over an extension field Fqδ , where δ is in O(`2 ).
In order to check Equation (6.2), we first need to compute π(Di ) and π 2 (Di ) which amounts
e 2 log q) operations in Fq . Then,
to raising the coordinates to the q-th power. The cost is in O(`
each Jacobian operation e 2 ) operations in the base field,
in the group generated by π(Di ) costs O(`
√
and we need O( `) of them to solve the discrete logarithm problem √ given by Equation (6.2).
e 2 ( ` + log q)) operations in Fq .
The overall cost of finding ki , once Di is known is therefore O(`
e 2 (`2 + log q)) operations in the base
Finally, the amount of work performed for each ` is O(`
field Fq . Summing up for all the primes, and taking into account the cost of the operations in
Fq , we obtain a global bit-complexity of O((log
e q)6 ).

6.2 Bounds for Algorithm 8

6.2.1 Bounds on the coefficients of ψ
The system of equations (6.1) giving σ1 , σ2 and σ3 in terms of a, b, c is homogeneous if we
put weight 1/2 to a, b, c and σ1 , weight 1 to q and σ2 , weight 3/2 to σ3 , and weight 0 to
µ0 , µ1 , and µ2 so any polynomial in a reduced Gröbner basis of the corresponding ideal will
have the same property. Computing such a Gröbner basis with the lexicographical ordering
a > b > c > σ1 > σ2 > σ3 > µ0 > µ1 > µ2 > q (we did this computation with the Magma
V2.23-4 software), we get a polynomial Ψc of degree 6 in c that does not involve a or b, and
which has the following form:
P5 (i)
Ψc (q, c, σ1 , σ2 , σ3 , µ0 , µ1 , µ2 ) = D(µ0 , µ1 , µ2 )3 c6 + i=0 ψc (q, σ1 , σ2 , σ3 , µ0 , µ1 , µ2 ) ci ,

where D(µ0 , µ1 , µ2 ) = −27 µ20 + 18 µ0 µ1 µ2 − 4 µ0 µ32 − 4 µ31 + µ21 µ22 is the discriminant of the
polynomial T 3 + µ2 T 2 + µ1 T + µ0 .
By computing Gröbner bases for other lexicographical orderings (with a > c > b > σ1 >
σ2 > σ3 > µ0 > µ1 > µ2 > q and b > c > a > σ1 > σ2 > σ3 > µ0 > µ1 > µ2 > q respectively),
we obtain that polynomials of the following form also belong to the ideal generated by the
polynomials in the system of equations (6.1):
P5 (i)
Ψb (q, b, σ1 , σ2 , σ3 , µ0 , µ1 , µ2 ) = D(µ0 , µ1 , µ2 )3 b6 + i=0 ψb (q, σ1 , σ2 , σ3 , µ0 , µ1 , µ2 ) bi ,
5 (i)
Ψa (q, a, σ1 , σ2 , σ3 , µ0 , µ1 , µ2 ) = D(µ0 , µ1 , µ2 )3 a6 + i=0 ψa (q, σ1 , σ2 , σ3 , µ0 , µ1 , µ2 ) ai .
P

(i) (i) (i)

The polynomials ψa , ψb and ψc are homogeneous of weighted degree 3 − i/2 with respect
to the grading given above.

Lemma 6.1. The absolute values of the coefficients a, b, c of ψ = a + bη + cη 2 are bounded above
by O(q 1/2 ).

Proof. First, we consider the equation Ψc = 0. We write c = ce q 1/2 , σ1 = σ

f1 q 1/2 , σ2 = σ
f2 q,
3/2 (i)
σ3 = σf3 q . Since ψc is homogeneous and has weighted degree 3 − i/2, there is a polynomial
(i)
θc (σf1 , σ
f2 , σ
f3 , µ0 , µ1 , µ2 ) such that

ψc(i) (q, σ1 , σ2 , σ3 , µ0 , µ1 , µ2 ) · ci = q 3 cei θc(i) (σ

f1 , σ
f2 , σ
f3 , µ0 , µ1 , µ2 ). (6.3)

Weil’s bounds imply that |σei | = O(1) for i ∈ {1, 2, 3}. Therefore, for all i ∈ {0, . . . , 5}, we obtain
(i)
that |θc (σf1 , σ f3 , µ0 , µ1 , µ2 )| = O(1). For fixed µ0 , µ1 , µ2 ∈ Q such that µ0 + µ1 T + µ2 T 2 + T 3
f2 , σ
90 Chapter 6. The case of genus-3 hyperelliptic curves with RM

is the minimal polynomial of a totally real algebraic number, the discriminant D(µ0 , µ1 , µ2 )
must be nonzero. Equations Ψc = 0 and (6.3) imply the following inequality:
5 (i)
X |θc (σf1 , σ
f2 , σ
f3 , µ0 , µ1 , µ2 )|
|ce|6 − |ce|i ≤ 0.
i=0
|D(µ0 , µ1 , µ2 )|3

Then |ce| must be smaller or equal to the largest root of this polynomial inequality, which can
itself be bounded, for instance, with Cauchy’s bound
( (i) )
|θc (σ
f1 , σ
f2 , σ
f3 , µ0 , µ1 , µ2 )|
|ce| ≤ 1 + max ,
0≤i≤5 |D(µ0 , µ1 , µ2 )|3

which shows that |ce| = O(1), and hence |c| = O(q 1/2 ). The proof for the bounds on |a| and |b|
are similar, using the equations Ψa = 0 and Ψb = 0.

6.2.2 Small elements in ideals of Z[η]

We first recall that we consider only primes ` that do not divide the discriminant of the minimal
polynomial of η (Condition (C2)). Hence, if Z[η] is not the maximal order of Q(η), this has no
consequence on the factorization properties of `.
Lemma 6.2. For any prime ` that splits completely in Z[η], each prime ideal pi above ` contains
a non-zero element αi of the form αi = ai + bi η + ci η 2 , where |ai |, |bi | and |ci | are integers in
O(`1/3 ), and the norm of αi is in O(`).
Proof. The coefficients of the elements of the ideal pi represented by polynomials in η form
a lattice. Applying Minkowski’s bound to this lattice, we obtain the existence of a non-zero
element αi = ai + bi η + ci η 2 , in pi for which the L2 -norm of (ai , bi , ci ) is in O(`1/3 ). From this
bound on the L2 -norm, we derive a bound on the L∞ -norm, and finally on the norm of αi as
an algebraic number. At each step, the constant hidden in the O() gets worse but still depends
only on Z[η].

For any given η, it is not difficult to make the constants in the O() fully explicit. We do
it in the particular case of Z[η7 ], with η7 = 2 cos(2π/7), which is the RM used in our practical
experiments. Since Z[η7 ] is a principal ring, a more direct approach leads to bounds for a
generator that are tighter than what would be obtained by a naive application of the previous
lemma.
Lemma 6.3. Every ideal pi of norm ` in Z[η7 ] has a generator αi of the form ai + bi η7 + ci η72 ,
where ai , bi , ci ∈ Z satisfy

|ai | < 2.415 · `1/3 ; |bi | < 1.850 · `1/3 ; |ci | < 1.764 · `1/3 .

Proof. By abuse of notation, we identify Q(η7 ) with the algebraic number field Q[X]/(X 3 +X 2 −
2X − 1) and we let σ1 , σ2 , σ3 be the three real embeddings of Q(η7 ) in R and let 1 = 1 − η72
and 2 = 1 + η7 be a pair of fundamental units. Let µi be a generator of pi . The logarithmic
embedding ϕ : x 7→ (log|σ1 (x)|, log|σ2 (x)|, log|σ3 (x)|) sends the set of generators of pi to the
lattice generated by ϕ(1 ) and ϕ(2 ) translated by ϕ(µi ). Solving a CVP for the projection
of ϕ(µi ) on the plane where the 3 coordinates sum-up to zero, we deduce a unit ξi such that
αi = ξi µi is a generator whose real embeddings are bounded by

|σ1 (αi )| ≤ 2.247 · `1/3 , |σ2 (αi )| ≤ 1.803 · `1/3 , |σ3 (αi )| ≤ 2.247 · `1/3 .
6.3. Computing kernels of endomorphisms 91

Writing αi = ai +bi η7 +ci η72 , the real embeddings can also be expressed as (σ1 (αi ), σ2 (αi ), σ3 (αi ))T =
V · (ai , bi , ci )T , where V is the Vandermonde matrix of (σ1 (η7 ), σ2 (η7 ), σ3 (η7 )). A numerical eval-
uation of its inverse allows to translate the bounds on σ1 (αi ), σ2 (αi ), σ3 (αi ) into the claimed
bounds on ai , bi , ci .

6.3 Computing kernels of endomorphisms

6.3.1 Modelling the kernel computation by a polynomial system

Let α be an explicit endomorphism of degree O(`2 ) on the Jacobian of C, which satisfies the
properties of Lemma 6.2. We want to compute a triangular polynomial system that describes
the kernel J[α] of α. This will provide us with a nice description of a subgroup of the `-
torsion on which we will be able to test the action of ψ = π + π ∨ and deduce a, b, c such that
ψ = a + bη + cη 2 mod `.
We first model J[α] by a system of polynomial equations that we will then put in triangular
form. To do so, we consider a generic divisor D = P1 + P2 + P3 − 3∞, where Pi is an affine
point of C of coordinates (xi , yi ). We then write α(D) = 0, i.e α(P1 − ∞) + α(P2 − ∞) =
−α(P3 − ∞). Generically, we expect each α(Pi − ∞) to be of weight 3, and we write hui , vi i
for its Mumford form. We derive our equations by computing the Mumford form hu12 , v12 i of
α(P1 − ∞) + α(P2 − ∞) and then writing coefficient-wise the conditions u12 = u3 and v12 = −v3 .
The case where the genericity conditions are not satisfied is discussed at the end of the section.
Similarly to the Schoof-Pila algorithm, we define polynomials — which are equivalent to
Cantor’s division polynomials — by the formulas

2 ˜ 2
X di (x1 , x2 , y1 , y2 ) X ẽi (x1 , x2 , y1 , y2 )
u12 (X) = X 3 + X i, v12 (X) = X i,
i=0 d˜3 (x1 , x2 ) i=0
ẽ3 (x1 , x2 )
2 2
X di (x3 ) i X ei (x3 ) i
u3 (X) = X 3 + X, v3 (X) = y3 X.
d (x )
i=0 3 3 i=0
e3 (x3 )

Lemma 6.4. For any i ∈ {1, 2, 3}, the degrees of d˜i , ẽi , di and ei are in O(`2/3 ).

Proof. Let us first remark that the d˜i ’s and ẽi ’s are obtained after adding two divisors hu1 , v1 i
and hu2 , v2 i such that the coefficients of the ui and vi are respectively the dj /d3 and yi ej /e3
evaluated at xi . Thus, since this application of the group law involves a number of operations
that is bounded independently of ` and q, the degree stays within a constant multiplicative
factor, which is captured by the O(). Therefore it is enough to prove the result for the di ’s and
ei ’s.
Since the endomorphism α satisfies the properties of Lemma 6.2, it is a linear combination of
1, η and η 2 with coefficients of size O(`1/3 ). Using the same argument about the group law, we
can further reduce our proof to the case where α = nη k , with k ∈ {0, 1, 2} and n an integer in
O(`1/3 ). But once again, η k does not depend on ` so that, provided we can prove that Cantor’s
n-division polynomials have degrees in O(n2 ), we have proven that nη k (P − ∞) = η k (n(P − ∞))
have coefficients whose degrees are in O(n2 ), and then so does α(P − ∞). This quadratic bound
on the degrees of Cantor’s division polynomials in genus 3 is precisely Theorem 4.14, whose
proof is done in Section 4.3.
92 Chapter 6. The case of genus-3 hyperelliptic curves with RM

6.3.2 Solving the system with resultants

Typical tools for solving a polynomial system are the F4 algorithm, methods based on geometric
resolution, or homotopy techniques. To obtain reasonable complexity bounds, they all require
some knowledge of the properties of the system, and this might be hard to prove. Since we have
a system in essentially 3 variables (in fact, there are six variables x1 , x2 , x3 , y1 , y2 , y3 , but the yi
variables can be directly eliminated by using the equation defining the curve), we prefer to stick
to an approach based on resultants. It ends up having a complexity that is quasi-quadratic in
the degree of the ideal, which is the best that can be hoped for anyway for all of the advanced
techniques, and the complexity analysis requires only elementary tools. A complication that can
occur with resultants is that Resx (f, g) is identically zero when f and g have a nonconstant GCD.
This is not a problem in our case since we can divide polynomials f and g by their GCD, by
factoring them at the cost of O(max(deg(f ), deg(g))ω ) field operations — where ω < 2.38 is the
exponent of linear algebra — using the bivariate recombination methods in [22] (the trivariate
case can be reduced to the bivariate case by using the techniques in [147, Sec. 21.2]). In what
follows, the complexities of computing the resultants are larger than O(max(deg(f ), deg(g))ω ),
so we can forget about this complication.
Following our modelling, the equality of the u-coordinates gives three equations

∀i ∈ {0, 1, 2}, d˜i (x1 , x2 , y1 , y2 )d3 (x3 ) = d˜3 (x1 , x2 )di (x3 ), (6.4)

of degree O(`2/3 ) in the xi ’s. By computing resultants with the equations yi2 = f (xi ), we derive
three equations Ei (x1 , x2 , x3 ) = 0 whose degrees are still in O(`2/3 ).
We then eliminate x1 by computing 3 trivariate resultants Ri (between the two equations
Ej with j 6= i). We get three equations Ri (x2 , x3 ) = 0 of degrees O(`4/3 ) within a complexity in
e 10/3 ) field operations, as proven in Proposition 2.40.
O(`
Then, we compute bivariate resultants Si (between the two equations Rj with j 6= i) to
eliminate x2 . From Proposition 2.39, we get three univariate equations Si (x3 ) = 0 of degree
bounded by O(`8/3 ) for a complexity in O(` e 4 ) field operations. And we compute the polynomial
S(x3 ) as the GCD of the Si (x3 ), which belongs to the ideal defined by our original system.
The bound on the degree of S is much larger than `2 − 1, the expected degree of the kernel.
Although we can expect the actual degree to be in O(`2 ), we need to add the constraints coming
from the v-coordinates to be able to prove it.
The polynomial system coming from v12 = −v3 has the same characteristics as the one
coming from the u-coordinates. Therefore, we can proceed in a similar way and deduce, at
e 4 ) operations another univariate polynomial S̃(x3 ) belonging to the ideal. Now,
a cost of O(`
since all the original equations have been taken into account all common roots of S and S̃ will
correspond to a solution of the original system for which we know that there are O(`2 ) solutions.
Therefore taking the squarefree part of the GCD of S and S̃ yields a polynomial of degree O(`2 ).
This univariate polynomial can be factored at a cost of O(` e 4 ) operations in Fq with standard
algorithms [54] (there exist asymptotically faster algorithms, but we already fit in our target
complexity). We then deal with each irreducible factor in turn, until one is found that leads
to a genuine solution of the original system. Let δ be the degree of such an irreducible factor
φ(x3 ). In the field extension Fqδ = Fq [x3 ]/(φ(x3 )), we have by construction a root x3 of φ. We
then solve again the original polynomial system where x3 is instantiated with this root. This
system is bivariate in x1 and x2 and there are O(1) solutions, that possibly live in another
finite extension Fqδ0 of Fqδ . Since the degrees of the bivariate polynomials are in O(`2/3 ), by
e 2 ) operations in F δ .
Proposition 2.39, this system solving costs O(` q
6.3. Computing kernels of endomorphisms 93

A solution obtained in this way must be checked, because it could come from a vanishing
denominator that has been cleared when constructing the system or from non-generic situations.
But given a set of candidate coordinates for a Di element of J[αi ], it is cheap to check that this
is indeed an element of the Jacobian and that it is killed by αi . Also, if αi is not a generator of
pi , it is necessary to check the order of Di : if this is a multiple of `, then multiplying Di by the
cofactor gives an order-` element. But it is also possible to get an unlucky element that is of a
small order coprime to `, and then we have to take another solution of the system.
Since an operation in Fqδ requires a number of operations in Fq that is quasi-linear in δ,
and since the sum of all the degrees δ of the irreducible factors of GCD(S, S̃) is in O(`2 ), the
e 4 ) operations in Fq to deduce a divisor Di in J[αi ].
amortized cost is O(`
Note that using the algorithm of Villard mentioned at the end of Section 2.3 for bivariate
resultants, the complexity is lowered as follows. First, to compute trivariate resultants of poly-
nomials whose degrees are bounded by d, we perform a Kronecker substitution and compute
bivariate resultants of equations of degrees dx ≤ d3 and dy ≤ d. Since d = `2/3 , we end up
with a complexity in O(`2/3(2−1/ω)+2+o(1) ), which is dominated by `3+1/9 . Once this is done, it
remains to compute bivariate resultants of equations with degrees in x and y both smaller than
2d2 . This yields a complexity in O(`4/3(2−1/ω) `4/3+o(1) ) field operations. This is dominated by
O(`3+5/9 ). It follows from our complexity analysis that the overall complexity of Algorithm 8 is
thus decreased by a factor (log q)4/9 at least.

6.3.3 Remarks

In Section 6.3, the algorithms work by evaluation / interpolation, which requires to have enough
elements in the base field. Were it not the case, we simply take a field extension Fqδ of Fq , that
will add a factor O(δ)
e to the complexity. The complexity of the algorithms will be polynomial
in the number of evaluation points, therefore, δ will be logarithmic in the final complexity, so
that the cost of taking a field extension will be hidden in the O()
e notation.

Another difficulty is that an evaluation / interpolation strategy assumes that the points of
evaluation are generic enough, so that all the degrees after evaluation are generic. This is again
guaranteed by taking a large enough base field. Still, the algorithm remains a Monte-Carlo one.
However, the ultimate goal is to construct kernel elements, which is an easily verified property.
Turning this into a Las Vegas algorithm can therefore be done with standard techniques.
Last but not least, our analysis assumes in the first place that the `-torsion elements are
generic in a rather strong sense, as in Definition 5.6. This is expected to be the case with
overwhelming probability, when the base field is large enough and the curve is taken at random
in a large family. However, to obtain a proven complexity we must also consider the cases where
there exist `-torsion elements that are non-generic. We follow the strategy of Section 5.4 where
another polynomial system is designed and solved for each non-generic situation, for instance
the fact that an `-torsion divisor is of weight less than 3, or that some points involved in the
modelling are not distinct while they generically are. We do not give all the details, but the
number of polynomial systems to consider is bounded by a constant, and each of these polynomial
systems describes a situation that is smaller than the generic one in the sense that it has either
less variables or a lower degree, so that the complexity bound is maintained.
94 Chapter 6. The case of genus-3 hyperelliptic curves with RM

6.4 Practical results

In this section, we compute the zeta function of a genus 3 hyperelliptic curve with explicit RM
defined over Fp with p = 264 − 59. To our knowledge the largest genus-3 computation that
had been achieved previously was the computation of the zeta function of a hyperelliptic curve
defined over Fp with p = 261 − 1, done by Sutherland [133] using generic group algorithms.
In order to evaluate the practicality of our algorithm, we have tested it on one of the families
of genus-3 hyperelliptic curves having explicit RM given in [138, Theorem 1]. Formulas for their
RM endomorphisms are described in [87]: for t 6= ±2, the curve Ct with equation

y 2 = x7 − 7x5 + 14x3 − 7x + t,

admits an endomorphism given in Mumford representation by

η7 (x, y) = hX 2 + 11 xX/2 + x2 − 16/9, yi.

The fact that this expression has degree 2 while one would generically expect a degree 3 is
no accident: it comes from the construction in [138] of the endomorphism as a sum of two
automorphisms on a double cover of the curve. We have η73 + η72 − 2η7 − 1 = 0, so that the ring
Z[η7 ] is isomorphic to the ring of integers Z[2 cos(2π/7)] of the real subfield of the cyclotomic
field Q(e2iπ/7 ). All the numerical data in this section have been obtained for the parameter
t = 42, on the prime field Fp with p = 264 − 59.
In our practical computations, the main differences with the theoretical description are the
following: we use Gröbner basis algorithms instead of resultants, we consider also small non-split
primes ` and small powers, and we finish the computation with a parallel collision search. The
source code for our experiments is available at https://members.loria.fr/SAbelard/RMg3.
tgz.

6.4.1 Retrieving modular information

Although the polynomial system resolution using resultants has a complexity in O(` e 4 ), the
real cost for small values of ` is already pretty large. In the resolution method described in
Section 2.3, each bivariate resultant is computed by evaluation / interpolation and hence requires
the computation of many univariate resultants. We illustrate this by counting the number of
univariate resultants to perform and their degrees for the main step of the resolution (the part
that reaches the peak complexity). We also measure the cost of such resultant computations
using the NTL 10.5.0 and FLINT 2.5.2 libraries, both linked against GMP 6, when the base
field is F264 −59 . These costs do not include the evaluation / interpolation steps which might also
be problematic for large instances, because they are hard to parallelize.

` #res Deg Cost (NTL) Cost (FLINT)

13 525M 16,000 1,850 days 735 days
29 12.8G 80,000 310,000 days 190,000 days

We were more successful with the direct approach using Gröbner bases that we now describe.
For computing the kernel of a given endomorphism, we computed a Gröbner basis of the sys-
tem (6.4) with some small modifications. First, we observe that the only occurrences of y1 and
y2 are within the monomial y1 y2 . Consequently, we can remove one variable by replacing each
occurrence of y1 y2 by a fresh variable y. Next, we need to make the system 0-dimensional by
encoding the fact that d3 (x3 ) and df3 (x1 , x2 ) are nonzero. This is done by introducing another
6.4. Practical results 95

fresh variable t and by adding the polynomial S(x1 , x2 , x3 )t−1 to the system, where S(x1 , x2 , x3 )
is the squarefree part of d3 (x3 )df3 (x1 , x2 ). Finally, it appears that each polynomial is symmetric
with respect to the transposition of the variables x1 and x2 . Consequently, we can rewrite the
equations using the symmetric polynomials s1 = x1 + x2 and s2 = x1 x2 . This divides by two
the degree in x1 and x2 of the equations. We end-up with a system in 5 variables.
The whole construction can be slightly modified to compute the pre-image of a given divisor
by the endomorphism: to model α(D) = Q − P∞ , we write D = P1 + P2 + P3 − 3P∞ and solve
for α(P1 − P∞ ) + α(P2 − P∞ ) = Q − P∞ − α(P3 − P∞ ). In that case, the variable y3 gets involved
in all the equations, so that we get a system in 6 variables.
For ` = 2, the 2-torsion elements are easily deduced from the factorization of f , and by
computing a pre-image of a 2-torsion divisor, we got a point in J[4] from which we could
deduce a, b, c mod 4. Dividing again by 2 was too costly, due to the fact that the 4-torsion
point was in an extension of degree 4. For ` = 3, which is an inert prime, we ran the kernel
computation for the multiplication-by-3 endomorphism, without using the RM property. The
norm being 27, this is the largest modular computation that we performed (and the most costly
in terms of time and memory). The prime ` = 7 ramifies in Z[η7 ] as the cube of the ideal
generated by α7 = −2 − η7 + η72 . The kernel of α7 can be computed but it yields only one
linear relation in a, b, c mod 7. Dividing the kernel elements by α7 would give more information,
but again, this computation did not finish due to the field extension in which the divisors
are defined. The first split prime is ` = 13. We use the following small generators: (13) =
(2 − η7 − 2η72 )(−2 + 2η7 + η72 )(3 + η7 − η72 ), which seem to produce the polynomial systems
with the smallest degrees. For instance, the apparently smaller element 1 + η72 of norm 13 yields
equations of much higher degrees 7, 71, 72, 73, 72. The next split prime is 29, which would maybe
have been feasible, but was not necessary for our setting. In the following table, we summarize
the data for these systems, that were obtained with Magma V2.23-4 on a Xeon E7-4850v3 at
2.20GHz, with 1.5 TB RAM.

mod `k #var degree of each eq. time memory a, b, c mod `k

2 — — — — 0, 0, 0
4 (inert2 ) 6 7, 7, 14, 15, 15, 10 1 min negl. 2, 2, 2
3 (inert) 5 7, 53, 54, 55, 26 14 days 140 GB 1, 2, 1
7 = p31 5 7, 35, 36, 37, 36 3.5h 6.6 GB a + 2b + 4c ≡ 2
13 = p1 p2 p3 5 7, 44, 45, 46, 52 3 × 3 days 41 GB 12, 10, 9
29 = p1 p2 p3 5 7, 92, 93, 94, 100 > 3×2 weeks > 0.8 TB —

6.4.2 Final collision search

The classical square-root-complexity search in genus 3 requires O(q) group operations [42]. For
RM curves, this can be improved by searching for the coefficients a, b, c of ψ = π + π ∨ in Z[η].
This readily yields a complexity in O(q 3/4 ), using the equation aD + bη(D) + cη 2 (D) = (q + 1)D,
that must be satisfied for any rational divisor D. While a baby-step giant-step approach is
immediate to design, it needs O(q 3/4 ) space and this is the bottleneck. A low-memory, parallel
version of this search can be obtained with the algorithm of [61], where the details are given
only for a 2-dimensional problem, while here this is a 3-dimensional problem. We explain below
how we modified this algorithm to fit our needs. Just like in [61], including some anterior
modular knowledge is straightforward: if a, b, c are known modulo m, the expected time is in
O(q 3/4 /m3/2 ).
This time, the search was performed in a cuboid instead of a rectangle. Contrary to the
96 Chapter 6. The case of genus-3 hyperelliptic curves with RM

general genus-2 case, this time the cuboid is not flat since a, b and c have the same order of
magnitude. Let us start by picking a random divisor D in J and set

KD = [a mod m]D + [b mod m]η(D) + [c mod m]η 2 (D).

As in Section 3.2.3, we look for a collision between two sets

T = {s1 mD + s2 mη(D) + s3 mη 2 (D) | (s1 , s2 , s3 ) ∈ [−B/m, B/m]3 },

and
W = {KD + s1 mD + s2 mη(D) + s3 mη 2 (D) | (s1 , s2 , s3 ) ∈ [−B/m, B/m]3 }.
From the relations (6.1) between the coefficients of ψ and the coefficients of χπ , one could
translate the Weil bounds into precise bounds in the coefficients a, b and c. Instead, we set an
√
ad hoc bound B = 5 q for their respective absolute values. Our choice was satisfactory and
we did not encounter any problem so we did not modify it, although fine tuning this parameter
would certainly reduce the average running time.
Each chain consists in a pseudo-random deterministic walk in either W or T that stops
whenever it encounters a distinguished point, which is the only information stored from each
chain. Indeed, the deterministic nature of the process guarantees that any collision between two
chains will propagate to their last point. This increases the running time compared to the baby-
step giant-step approach but allows for negligible memory requirements, as explained in 3.2.3.
While the probability pD of being distinguished is an important parameter, the distinguishing
feature itself is not. For instance, we say that an element is distinguished if the b− log2 pD c bits
of low weight of its Mumford representation are equal to 0. By the birthday paradox, we expect
a collision to be found after browsing through (2B/m)3/2 points in the searchspace. Denoting
by C the number of chains, we therefore expect each chain to be of length (2B/m)3/2 /C, and
since each chain stops whenever it hits a distinguished element, pD is precisely the inverse of
this quantity. In our experiments, we set pD = 50000(B/m)−3/2 , thus expecting the number of
chains to be about 140000 before a collision occurs. Recall that the number of chains must be
small enough to keep the memory requirements reasonable, but large enough to avoid taking
too much time.
To design the deterministic walks, we start each chain by an element of either T or W defined
by a triple (s1 , s2 , s3 ) taken uniformly at random in [−B/m, B/m]3 . Then, given a divisor D̃ in
a chain, the next one is computed as D̃ + Oh(D̃) , where the O’s are a set of 120 precomputed
offsets and h a hash function mapping D̃ into a triple (b1 , b2 , i) ∈ {0, 1}2 × {1, 2, . . . , 30}. The
offset corresponding to that triple is

αi mD + (−1)b1 βi mη(D) + (−1)b2 γi mη 2 (D),

where αi , βi and γi are integers respectively taken uniformly at random in {1, 2, . . . , 2Li } and
then fixed during the whole search. The Li ’s are chosen to reduce the risk of a chain exiting the
cuboid and considering points on which collisions are impossible. This could actually even lead
to a neverending chain which is why some bound can be set to discard any chain whose length
is much longer than expected, but a convenient choice for the Li ’s make this extremely unlikely.
Our practical choice followed the genus-2 case and set the Li ’s such that, on average, each chain
terminates on a point whose coordinates are ten times smaller than the size of the cuboid. In
the first direction, our offset is always positive and the expected length of a chain is 1/pD ,
so that the expected distance in the first direction is L1 /pD and we choose L1 = 2BpD /10.
For the two other directions, the offsets have changing signs so that we bound the distance
6.4. Practical results 97

p √
reached using the central limit theorem. This yields an expected distance in 2 2/3πL2 / pD
as in the genus-2 case. This is not surprising because since we study the problem dimension by
dimension, we always considerp one-dimensional random walks no matter the dimension of the
searchspace. Approximating 2 2/3π by 9/10 and dividing by 10, we choose L2 and L3 both
√
equal to 2B pD /10.
We wrote a dedicated C implementation with a few lines of assembly to speed-up the addi-
tions and multiplications in Fp , taking advantage of the special form of p. This implementation
performs 10.7M operations in the Jacobian per second using 32 (hyperthreaded) threads of a 16-
core bi-Xeon E5-2650 at 2 GHz. We used the knowledge of ψ modulo 156 but not of the known
relation modulo 7 for simplicity (there is no obstruction to using it and saving an additional
71/2 factor).
After computing about 190,000 chains of average length 32,000,000, we got a collision, from
which we deduced

ψ = 2551309006 + 2431319810 η7 − 847267802 η72 ,

and the coefficients of the characteristic polynomial χπ of the Frobenius are then

σ1 = 986268198, σ2 = 35389772484832465583, σ3 = 10956052862104236818770212244.

The number of group operations that were done is slightly less than 43 (p3/4 /1563/2 ). This
factor 43 is close to the average that we observed in our numerous experiments with smaller
sizes. Scaled on a single (physical) core, we can estimate the cost of this collision search to be
105 core-days.
98 Chapter 6. The case of genus-3 hyperelliptic curves with RM
 Chapter 7

Counting points on hyperelliptic

curves with explicit RM

In this chapter, we study the benefits of real multiplication in arbitrary genus. We extend the
process of Chapter 6 and Section 3.1.2 for (families of) hyperelliptic curves with RM by an order
Z[η]. For primes ` that split into gi=1 pi in Z[η], we split J[`] into a direct sum of g subspaces
Q

J[pi ] isomorphic to (Z /` Z)2 . One can therefore expect that for RM curves, Algorithm 6 detailed
in Chapter 5 can be adapted
 to find non-zero
  of J[pi ] instead of J[`] with a complexity
elements
bound in Oη (log q) O(1) instead of Og (log q) O(g) . Note that we do not use the Og ()-notation
because, as in Chapter 6, there is an additional dependency in η. Since g is nothing more than
the degree of the algebraic number η, we replace the Og ()-notation by the Oη ()-notation which
takes into account both dependencies on g and η.
Using a theoretical machinery similar to that of Chapter 5, we will prove that it is indeed the
case. However, we warn the reader that this complexity is still exponential in g. Even though
each of the ideals J[αi ] has degree independent of g, we model them by polynomial systems whose
2
multihomogeneous Bézout bounds involve a combinatorial factor of the form g g+g . Since the

complexity of the geometric resolution algorithm is quadratic in the multihomogeneous Bézout

bound, this exponential factor also appears in the overall complexity of our algorithm.

Organization. In Section 7.1, we give an overview of our point-counting algorithm, along with
an example of families of hyperelliptic curves of arbitrary high genus with RM by a real subfield of
a cyclotomic field. In particular, we prove a bound on the size and number of primes ` to consider
in our algorithm. Section 7.2 focuses on the main primitive of our algorithm: the computation
of a non-zero element in the kernel of an endomorphism α whose degree is a small multiple of `2 .
This section adapts methods and results of Chapter 5 to design structured polynomial systems
whose solution sets are subsets of J[α]. Section 7.3 concludes on the complexity of solving these
systems, and on the overall complexity of our point-counting algorithm. We also present an
analysis on the exponent of g in the final complexity, investigating the various places where
exponential factors may occur and how to avoid them when it is possible.

7.1 Overview
The main result of this chapter can be summarized by the following theorem, in which we give
more precision on the notation Oη (logc q) for our complexity result, and make the dependency

99
100 Chapter 7. Counting points on hyperelliptic curves with explicit RM

in η explicit. In Section 7.3, we also bound c by 8 and conjecture that it should be 6. Note that
whenever we give a bound with an explicit constant, we can no longer hide the polylogarithmic
factor in the exponent, so we use the notation O e η () to hide both factors depending only on η
and factors that are polylogarithmic in q.
Theorem 7.1. For any g and any η ∈ Q such that Q(η) is a totally-real number field of degree g,
there exists an explicitly computable c(η) > 0 such that there is an integer q0 (g, η) such that for
all prime power q = pn larger than q0 (g, η) with p ≥ (log q)c(η) and for all genus-g hyperelliptic
curves C with explicit RM by Z[η] defined over Fq , the local zeta function of C can be computed
with a probabilistic algorithm in expected time bounded by (log q)c(η) .

7.1.1 Families of RM curves

We present one-dimensional families of hyperelliptic curves from [138], constructed via cyclo-
tomic covers. They have an affine model Cn,t : Y 2 = Dn (X)+t, where t is a parameter and Dn is
the n-th Dickson polynomial with parameter 1 defined inductively by D0 (X) = 2, D1 (X) = X,
and
Dn (X) = XDn−1 (X) − Dn−2 (X).
Since Dn (X) has degree n, setting n = 2g + 1 for odd n yields a one-dimensional family Cn,t
of genus g hyperelliptic curves given by an odd-degree Weierstrass model. Their Jacobians
all have an explicit endomorphism η, and when n is prime, Proposition 2 of [87] shows that
Z[η] ∼
= Z[ζn + ζn−1 ], where ζn is a primitive n-th root of unity over Q. Another family based on
Artin-Schreier covering is detailed in the same paper but these curves have genus (p−1)/2 where
p is the characteristic of the base field, so that our complexity study using the Oη () notation
would be pointless in that case. Since g becomes much larger than log p in that case, it would
be more efficient to use p-adic algorithms anyway.
Let C be a hyperelliptic curve of genus in the family C2g+1,t . In [87], Kohel and Smith
compute formulas for the Mumford form of η ((x, y) − P∞ ), where (x, y) is the generic point on
C. These formulas are given explicitly for some examples in genus 2 and 3, and an algorithm [87,
Algorithm 5] is presented to compute them for any C. This algorithm has a time complexity in
O(g 2 ) and requires to store O(g 3 ) field elements. Thus, given a curve from that family as input,
an explicit endomorphism of its Jacobian can be computed once and for all in O(g 3 log q) time
and space complexity, which is negligible compared to the cost of counting points on the curve.

7.1.2 The characteristic equation

As in genus 3, let us consider ψ = π + π ∨ and recall that ψ ∈ Q[η]. We still have ψπ = π 2 + q
and once again, we test this equation to determine ψ instead of the characteristic equation of
π. The link between ψ and π needs to be made explicit, which is the aim of the present section.
Since χπ is a Weil polynomial, we can write χπ (X) = gi=0 (−1)i σi (X 2g−i + q g−i X i ), with
P

σ0 = 1 and the convention that σg is actually twice smaller than the g-th coefficient of χπ . By
the Cayley-Hamilton theorem, we have q −g (π ∨ )g χπ (π) = 0. Using the fact that ππ ∨ = q, we
rewrite that as g
(−1)g−i σg−i (π i + (π ∨ )i ) = 0.
X

i=0

Our plan is to compute χπ mod ` by determining ψ . Let us write ψ = g−1 i

P
i=0 ai η , the goal
of the section is to prove bounds on the coefficients ai , so that we can estimate the number
and maximal size of primes ` required to compute ψ without ambiguity. Note that ψ is in the
7.1. Overview 101

maximal order of Q(η), but not necessarily in Z[η]. However, as in the genus-3 case, Z[η] has
finite index ∆ in the maximal order and the possible common denominator of the ai ’s has to
divide ∆. This denominator entails that additional primes may be required to fully determine
ψ, however ∆ depends only on η so that it will disappear in the Oη -notation of our complexity
estimates. Therefore, we do not detail further this subtlety and assume for simplicity that the
√
ai ’s are integers, which we wish to bound by Oη ( q).
Let us first express the quantities π i + (π ∨ )i in terms of powers of ψ as a first step towards
expressing the σi ’s as functions of the ai ’s.
Lemma 7.2. For any i ∈ {1, . . . , g}, there exist integers (αi,j )0≤j<i such that αi,j = O(q (i−j)/2 )
and
i−1
∨ i
X
i i
π + (π ) = ψ + αi,j ψ j .
j=0

Proof. The statement holds for i = 1 with α1,0 by the definition of ψ. For i = 2, we have
ψ 2 = π 2 + (π ∨ )2 + 2ππ ∨ , so that we have the result with α2,0 = −2q and α2,1 = 0.
In this proof, we set the convention αi,i = 1 to simplify our recurrence relations.
Let us now assume the lemma holds for any positive integer no greater than a certain i. We
therefore have
 
i−1
ψ i+1 = (π + π ∨ )ψ i = (π + π ∨ ) (π i + (π ∨ )i ) −
X
αi,j ψ j  .
j=0

The first term is equal to π i+1 + (π ∨ )i+1 + q(π i−1 + (π ∨ )i−1 ) so that we can use the lemma once
again for i − 1 and get
i−1
ψ i+1 = π i+1 + (π ∨ )i+1 − αi,i−1 ψ i + qαi−1,0 +
X
(qαi−1,j − αi,j−1 )ψ j .
j=1

Thus, we have computed the αi+1,j given by




 αi,i−1 if j = i,
αi+1,j = −qα i−1,0 if j = 0,

i,j−1 − qαi−1,j

 α else.

Let us now study the order of magnitude of the αi+1,j : from the recurrence hypothesis on both
√
i and i − 1, αi,i−1 = αi+1,i is in O( q), αi−1,0 is in O(q (i−1)/2 ) so that αi+1,0 is in O(q (i+1)/2 ),
and both qαi−1,j and αi,j−1 are in O(q (i+1−j)/2 ), which proves the result for any other αi+1,j .
By induction, the lemma is proven.

Note that our O-notation in the previous statement and proof can be a bit misleading as there
may not be an absolute constant bounding all the αi,j /q (i−j)/2 . However, from the recurrence
relation between the ai,j ’s, one sees that each αi,j is equal to q (i−j)/2 plus an error term that
is in Oη (q (i−j−1)/2 ) and at worst quadratic in g, hence the error term is negligible compared to
q (i−j)/2 .
Proposition 7.3. Let the ai ’s be the coefficients of ψ in the basis (1, η, . . . , η g−1 ) and σi be the
i-th coefficient of χπ , or half this coefficient if i = g. Then χπ is uniquely determined by the ai ’s
and there exists Cη > 0 depending only on g and η such that for any i ∈ {0, . . . , g − 1}, we have
√
|ai | ≤ Cη q.
102 Chapter 7. Counting points on hyperelliptic curves with explicit RM

Proof. Using Lemma 7.2 for any i ∈ {1, . . . , g} and setting αi,i = 1, we have
g
X i
X g
X g
X
(−1)g−i σg−i αi,j ψ j = ψj (−1)g−i αi,j σg−i = 0.
i=0 j=0 j=0 i=j

Let us define χψ (X) = X g + sg−1 X g−1 + · · · + s0 with si = gi=j (−1)g−i αi,j σg−i . Invoking
P

the Weil conjectures for the σg−i ’s and Lemma 7.2 for the αi,j , one concludes that each si
is in O(q (g−i)/2 ). Furthermore, the expressions of the si ’s in terms of the σi ’s form a linear
triangular system whose determinant equals 1, so that there is an efficiently computable one-to-
one correspondence between χψ and χπ .
Let us now make explicit the link between the coordinates ai of ψ = g−1 i
P
Pg−1 i=0 ai η and the
coefficients si of χψ . For instance, sg−1 = − Tr(ψ) = − i=0 ai Tr(η i ). To get the other relations,
let us now order the g conjugates of η (possibly in the Galois-closure of Q(η)), numbering them
Pg−1
from η1 to ηg , and proceed to the linear change of variables ψk = i=0 ai ηki for any k in
{1, . . . , g}. The matrix associated to this linear transformation is the Vandermonde matrix of
the conjugates ηk ’s. This matrix is invertible because η is separable so that the ηi are all distinct
reals.
Note that χψ is a degree-g monic polynomial vanishing on ψ, and it is therefore its char-
acteristic polynomial. Since the ψk are exactly the real roots (possibly in the Galois-closure of
Q(η)) of χψ , by Vieta’s formula they satisfy the g equations

sg−i = (−1)i Si (ψ1 , . . . , ψg ) for 1 ≤ i ≤ g,

where the Si ’s are the elementary symmetric polynomials in g variables. Thus, once the ai ’s are
known, the values for ψ and its conjugates are known and a unique value for each si is deduced.
Furthermore, the Fujiwara bounds from [51] imply that for any k ∈ {1, . . . , g} we have
 
|ψk | ≤ 2 max |sg−k |1/k .
0≤k≤g

√ √
We already know that |sg−k | = O( q k ), so we deduce that the |ψk | are in O( q). Then,
√
inverting the linear change of variable, we prove that the ai are also in Oη ( q) since the matrix
norm of the inverse of the Vandermonde matrix only depends on η.

Our algorithm is based on determining the ai ’s modulo ` for sufficiently many ` until they
are known without ambiguity and we can deduce χπ . While the Weil bounds on the σi ’s are
√
enough for our purpose, we have proven that the ai ’s are in Og ( q) as in the genus-3 case. The
next section details the process of recovering such modular information on the ai ’s.

7.1.3 Overview of our algorithm

The general RM point counting algorithm is Algorithm 9. As mentioned above, we want to
compute the coefficients a0 , . . . , ag−1 of the endomorphism ψ. More precisely, we compute their
values modulo sufficiently many totally-split primes ` until we can deduce their values from
the bounds of Prop 7.3 and the Chinese Remainder Theorem. Then, the coefficients of χπ are
deduced from the ai ’s.
We now explain how the algorithm works for a given split `. First its decomposition as a
product of prime ideals ` Z[η] = p1 · · · pg is computed, and for each prime ideal pi , a non-zero
element αi in pi is found with a small representation as in Lemma 7.4 below. In fact, pi is not
necessarily principal and αi need not generate pi . The kernel of αi is denoted by J[αi ] and it
7.1. Overview 103

input : q an odd prime power, and f ∈ Fq [X] a monic squarefree polynomial of degree
2g + 1 such that the curve Y 2 = f (X) has explicit RM by Z[η].
output: The characteristic polynomial χπ ∈ Z[T ] of the Frobenius endomorphism on
the Jacobian J of the curve.
w ← 1;
Define Cg as in Prop. 7.3;
√
while w ≤ 2 ∆Cg q + 1 do
Pick the next prime ` that satisfies conditions (C1) to (C4);
Compute the ideal decomposition ` Z[η] = p1 · · · pg , corresponding to the eigenvalues
λ1 , . . . , λg of η in J[`] ;
for i ← 1 to g do
Compute a small element αi of pi as in Lemma 7.4;
Compute a non-zero element Di of order ` in J[αi ] ;
Find the unique ki ∈ Z /` Z such that ki π(Di ) = π 2 (Di ) + qDi ;
end
Find the unique tuple (a0 , . . . , ag−1 ) in (Z /` Z)g such that g−1 j
P
j=0 aj λi = ki , for i in
{1, . . . , g} ;
w ← w · `;
end
Reconstruct (a0 , . . . , ag−1 ) using the Chinese Remainder Theorem ;
Deduce χπ from ψ.
Algorithm 9: Overview of our RM point-counting algorithm

contains a subgroup Gi isomorphic to Z /` Z × Z /` Z, since the norm of αi is a multiple of `.

The two-element representation (`, η − λi ) of the ideal pi implies that λi is an eigenvalue of η
viewed as an endomorphism of J[`] ∼ = (Z/`Z)2g .
On Gi ⊂ J[αi ], the endomorphism η acts as the multiplication by λi . Therefore, the endo-
morphism ψ = g−1 i
P
i=0 ai η also acts as a scalar multiplication on this 2-dimensional space, and
we write ki ∈ Z /` Z the corresponding eigenvalue: for any Di in Gi , we have ψ(Di ) = ki Di . On
the other hand, from the definition of ψ, it follows that ψπ = π 2 + q. Therefore, if such a Di is
known, we can test which value of ki ∈ Z /` Z satisfies

ki π(Di ) = π 2 (Di ) + qDi . (7.1)

Since ` is a prime and Di is of order exactly `, this is also the case for π(Di ). Finding ki can then
be seen as a discrete logarithm problem in the subgroup of order ` generated by π(Di ); hence
the solution is unique. Equating the two expressions for ψ, we get explicit relations between the
aj ’s modulo `:
g−1
aj λji ≡ ki mod `.
X

j=0

Therefore we have a linear system of g equations in g unknowns, the determinant of which is

the Vandermonde determinant of the λi , which are distinct by hypothesis. Hence the system
can be solved and it has a unique solution modulo `.
It remains to show how to construct a divisor Di in Gi , i.e. an element of order ` in the
kernel J[αi ]. Since an explicit expression of η as an endomorphism of the Jacobian of C is known,
an explicit expression can be deduced for αi , using the explicit group law. The coordinates of
104 Chapter 7. Counting points on hyperelliptic curves with explicit RM

the elements of this kernel are solutions of a polynomial system that can be directly derived
from this expression of αi , using a modelling similar to that of Chapter 5. Likewise, we use the
geometric resolution algorithm to find the solutions of this system, perhaps in a finite extension
of the base field, from which divisors in J[αi ] can be constructed. Multiplying by the appropriate
cofactor, we can reach all the elements of Gi ; but we stop as soon as we get a non-trivial one.
We summarize the conditions that must be satisfied by the primes ` that we work with:

(C1) ` must be different from the characteristic of the base field;

(C2) ` must be coprime to the discriminant of the minimal polynomial of η;

(C3) there must exist αi ∈ pi as in Lemma 7.4 below with norm non-divisible by `3 for i ∈
{1, . . . g};

(C4) the ideal ` Z[η] must split completely.

The first 3 conditions eliminate only a finite number of `’s that depends only on η. The condition
(C3) implies that there is a unique subgroup Gi of order `2 in J[αi ].
Given a genus-g curve C with RM by Z[η], by Chebotarev’s density theorem, the proportion
of primes ` satisfying the last condition is at least 1/# Gal(Q(η)/ Q), which is bounded below by
1/(g!). To count points on C, we need to find L a set of primes satisfying all the above conditions
Q √
and such that `∈L ` > 2∆Cη q. By the prime number theorem, both the number and size of
the primes contained in L are in O((g!) log(Cg q)). In some particular cases, the proportion of
“nice” primes may be much larger: for instance when the RM field is the totally real subfield of
a cyclotomic field. In the field Q(ζn + ζn−1 ), a prime ` totally splits if and only if ` ≡ ±1 mod n,
and therefore condition (C4) is satisfied by a proportion of primes equal to 2/(n − 1) = 1/g.

Lemma 7.4. For any prime ` that splits completely in Z[η], each prime ideal p above ` contains
α = ig−1 i
i=0 αi η , where the |αi | are integers smaller than
P
a non-zero element α of the formh
∆1/g `1/g , where ∆ is the index OQ(η) : Z[η] .

Proof. The coefficients of the elements of the ideal p represented by polynomials in η form a
lattice L of dimension g. In Z[η], its volume is the norm of p, i.e. `. Thus, its actual volume
in Rg is `∆. Let us consider C = {x ∈ Rg | ||x||∞ ≤ ∆1/g `1/g }. The volume of the convex
C is 2g ∆`. Since g is the dimension of L and ∆` its volume, Minkowski’s theorem guarantees
the existence of a non-zero element v of L belonging to C. By definition, v = g−1 i
P
i=0 vi η is an
element of p whose coordinates vi ’s are integers of absolute values bounded by ∆1/g `1/g , which
concludes the proof.

Since we know it exists, given one of the ideals pi , we can find αi a small element of pi as
in Lemma 7.4 by exhaustive search in at most 2g ∆` operations in Z[η]. Note that there is an
extensive litterature on finding short vectors in a lattice of dimension d, motivated for instance
by cryptographic
√
applications. An example is the quantum algorithm of [39] which computes
a 2 e d)
O( -approximation of the shortest non-zero vector in time polynomial in d. Restricting
to classical algorithms, the best option in general is the BKZ algorithm [126] that computes a
e α) e 1−α )
2O(d -approximation in time 2O(d , for any α ∈ [0, 1]. In our case however, the existence of
a very short vector is already known and, more importantly, the factor 2g due to the dimension
is acceptable since it vanishes in the Oη -notation.
7.2. Modelling kernels of endomorphisms 105

7.2 Modelling kernels of endomorphisms

Let α be an explicit endomorphism of degree O(`2 ) on the Jacobian of C, which satisfies the
properties of Lemma 7.4. We want to compute a polynomial system that describes the kernel
J[α] of α, and then solve it. The resultant-based approach of Chapter 6 cannot be used as the
degrees are squared each time we eliminate a variable, causing an exponential dependency in
g in the exponent of `. Instead, we use the modelling techniques from Chapter 5, where the
endomorphism α replaces the multiplication by `. This time, the g variables of large degrees
have degrees in Oη (`3/g ) instead of Oη (`3 ) so that the final complexity bound for computing the
kernel α is in Oη (`c ), with c an absolute constant.
The main change between this section and Sections 5.3 and 5.4 is that the di and ei no
longer denote `-division but α-division polynomials, and the polynomials uj and vj intervening
in the Mumford representation of the candidate kernel element are modified accordingly. The
structure of our modelling is very similar but require some adaptations at various places, which
is the reason why we repeat the analysis in the generic case. In the non-generic case, we restate
the main results of Section 5.4 but only detail the parts requiring adjustments.

7.2.1 The generic case

Let us first recall the definition of Cantor’s `-division polynomials, the coefficients of the poly-
nomials δ` (X) and ε` (X) such that, for (x, y) a generic point of the curve and ` > g, we have
x−X x−X
    
` ((x, y) − P∞ ) = δ` , ε` .
4y 2 4y 2
We recall Theorem 4.13 proven in Chapter 4:
For any integer ` > g, the polynomial δ` (X) of degree g in X has coefficients in Fq [x]
whose degrees in x are bounded by g`3 /3 + Og (`2 ); the polynomial ε` (X)/y has coefficients in
Fq (x) such that the degrees of the numerators and the denominators have degrees bounded by
2g`3 /3 + Og (`2 ). Furthermore, the roots of the denominators are roots of the leading coefficient
of δ` (X).
These polynomials describe the multiplication by `, but for our purpose we need to define
the α-division polynomials di and ei such that, denoting by P = (x, y) the generic point of C,
the non-normalized Mumford form of α(P − P∞ ) is equal to
* g g−1 +
X
i
X ei (x) i
di (x)X , y X .
i=0 i=0
eg (x)
Pg−1
By Lemma 7.4, we know that α = i=0 αi η i with |αi | = Oη (`1/g ). Since the degrees of the
η i (P − P∞ ) do not depend on `, by Theorem 4.13 applied to Cantor’s αi -division polynomials
we prove that the degrees of the di ’s and ei ’s are in Oη (`3/g ).
Definition 7.5. In what follows, we will say that an element of J is α-generic if it has weight
g and the corresponding reduced divisor gi=1 (Pi − P∞ ) satisfies the following two properties:
P

• For any i, the u-coordinate of the divisor α(Pi − P∞ ) in Mumford form has degree g;

• For any i 6= j, the u-coordinates of the divisors α(Pi − P∞ ) and α(Pj − P∞ ) are coprime.
This implies that if an affine point P occurs in the support of a α(Pi − P∞ ) then neither P
nor −P appears in the support of another α(Pj − P∞ ).
106 Chapter 7. Counting points on hyperelliptic curves with explicit RM

Let D = gi=1 (Pi − P∞ ) be an α-generic divisor in J. We shall consider a system equivalent

P

to α(D) = 0 but let us first introduce some notation. For each point Pi = (xi , yi ) in the support
of D, we denote hui , vi i the Mumford form of α(Pi − P∞ ) and (aij , bij )1≤j≤g the coordinates of
the g points in its support counted with multiplicities, which means that for any i the g roots
of ui are exactly the aij , and that for any j, bij = vi (aij ).
Proposition 7.6. We can model the set of generic α-division elements as the solution set of
a bihomogeneous polynomial system consisting of O(g 2 ) equations in Fq [X1 , . . . , Xg , Y1 , . . . , Yny ]
such that ny = O(g 2 ) and the degrees in the Xi ’s and Yj ’s are respectively in Oη (`3/g ) and Oη (1).
Proof. Following the modelling of Section 5.3, we have α(D) = 0 if and only if the sum of the
divisors gi=1 α(Pi − P∞ ) is a principal divisor. The only pole is at infinity, so this is equivalent
P

to the existence of a non-zero function ϕ ∈ Fq (C) of the form P (X) + Y Q(X) with P and Q
two polynomials such that the g 2 points (aij , bij ) are the zeros of ϕ, with multiplicities. Since
we want ϕ to have g 2 affine points of intersection with the curve C (once again, counted with
multiplicities), the polynomial ResY (Y 2 − f, P + Y Q) = P 2 − f Q2 must have degree g 2 which
yields 2 deg(P ) ≤ g 2 and 2 deg(Q) ≤ g 2 − 2g − 1. Exactly one of those two bounds is even (it
depends on the parity of g), and for this particular bound, the inequality must be an equality,
otherwise the degree of the resultant would not be g 2 . Since the function ϕ is defined up to a
multiplicative constant, we can normalize it so that the polynomial P 2 + f Q2 is monic, which
is equivalent to enforce that either P or Q is monic depending on the parity of g.
For a fixed i ∈ [1, g], requiring the (aij , bij ) to be zeros of ϕ amounts to asking for the aij
to be roots of P (X) + Q(X)vi (X), with multiplicities. Since the aij are by definition the roots
of the ui , α(D) = 0 is equivalent to g congruence relations P + Qvi ≡ 0 mod ui . Thus, for any
α-generic divisor, α(D) = 0 is equivalent to the existence of P and Q satisfying the above g
congruence relations.
The variables are the coefficients of P and Q, as well as the xi and yi . With the degree
conditions and the normalization, we have g 2 − g variables coming from P and Q. Adding
the 2g variables xi and yi , we get a total of g 2 + g variables. Each one of the g congruence
relations amounts to g equations providing a total of g 2 conditions on the coefficients of P and
Q. The fact that the (xi , yi ) are points of the curve yields the g additional equations yi2 = f (xi ).
Finally, we have to enforce the α-genericity of the solutions, which can be done by requiring
that i dg (xi )eg (xi ) i<j Res(ui , uj ) 6= 0. Note that we do not extend Theorem 4.13 but instead
Q Q

add the non-vanishing condition for the denominator of the v-coordinate of α(D). Still, we get
a polynomial system with g 2 + g equations in g 2 + g variables, together with an inequality.
We now estimate the degrees to which the variables occur in the equations. Each congruence
relation is obtained by reducing P +Qvi , which is a polynomial of degree O(g 2 ) in X, by ui which
is of degree g. We can do it by repeatedly replacing X g by − j<g (dj (xi )/dg (xi ))X j , which we
P

will have to do at most O(g 2 ) times. Since the dj have degree in Oη (`3/g ) in xi , the fully reduced
polynomial will have coefficients that are fractions for which the degrees of the numerators and
of the denominators are at most Oη (`3/g ) in the xi variables. In these equations, the degree in
the yi variables and in the variables for the coefficients of P and Q is 1. The degrees in xi and
yi in the curve equations are 2g + 1 and 2 respectively.
It remains to study the degree of the inequality. Each resultant is the determinant of a 2g×2g
Sylvester matrix whose coefficients are the di , which have degrees bounded by Oη (`3/g ). Since
for any i there are exactly g resultants involving xi in the product, the degree of this inequality
in any xi is in Oη (`3/g ), and it does not involve the other variables. In order to be able to
use Proposition 5.3, we must model this inequality by an equation, which is done classically by
introducing a new variable T and by using the equation T · i dg (xi )eg (xi ) i<j Res(ui , uj ) = 1.
Q Q
7.2. Modelling kernels of endomorphisms 107

To conclude, we have a polynomial system with two blocks of variables: the g variables xi
on the one hand and the g 2 − g variables coming from the coefficients of P and Q, along with
the g variables yi on the other hand. The degree of the equations in the first block of variables
grows cubically in `1/g , while the degree in the other block of variables depends only on g (and
η).

7.2.2 Non-generic kernel elements

As in Section 5.3, apart from the neutral element, we expect to capture the whole kernel of the
endomorphism α by using the modelling of Section 7.2.1. Contrary to Chapter 5, Algorithm 9
does not require us to find a basis of J[α] because the determination of the ki ’s does only
require a single non-zero element in each J[αi ]. Thus, a study of non-generic elements in J[α]
is necessary only if there is no α-generic element in J[α]. Such a case happens if and only if
the polynomial gi=1 dg (xi )eg (xi ) i6=j Res(ui , uj ) in the variables x1 , . . . , xg vanishes on J[α]. It
Q Q

seems very unlikely that the whole set J[α] lives in such a hypersurface, and if it happens, one
can discard the ` for which we fail to find an α-generic element. Although it seems even more
unlikely that this situation could happen for sufficiently many ` so as to threaten the validity
of our complexity bound, we are far from a proven statement and do not exclude it might be
possible to design a highly non-generic curve providing a counterexample.
Therefore, we follow the non-genericity analysis of Section 5.4 except that we consider ui
and vi defined as the Mumford form of α(Pi − P∞ ) instead of `(Pi − P∞ ). Let us briefly review
the non-generic situations that one can encounter, following Section 5.4.1 and keeping the same
numbering.

Case 1: Modelling a kernel element of weight w < g. We write D = w i=1 (Pi − P∞ )

P

and look for a ϕ = P (X) + Y Q(X) vanishing at each point of each reduced divisor α(Pi − P∞ ).
This is similar to the Case 1 of Section 5.4.1.

Case 2: Modelling a kernel element with multiple points. It may happen that the
element we are looking for is D = w i=1 (Pi − P∞ ) but not all the Pi ’s are distinct. In that
P
Ps
case, we rewrite it as D = j=1 λj (Pj − P∞ ) such that the Pj ’s are distinct and look for a
ϕ = P (X) + Y Q(X) vanishing at each point of each reduced divisor λj α(Pj − P∞ ). Apart from
the modification of ui and vi , the modelling is identical to that of Chapter 5.

Case 4: Modelling a kernel element after reduction. Even if all the α(Pi − P∞ ) had
full weight, there still may be less than g 2 points in the union of their supports due to possible
cancellations of points appearing in the supports of several α(Pi − P∞ ) with different signs.
Exactly as in Section 5.4.1, if P appears within α(Pi − P∞ ) and α(Pj − P∞ ) with respective
multiplicities νi and νj of opposite signs, this is modelled by ensuring that the corresponding
ui , uj , and vi + vj share a common factor (X − ξ)ν where ν = max(|νi |, |νj |). In that case, we
look for ϕ(X, Y ) = (X − ξ)ν (Pe (X) + Y Q(X)),
e with Pe coprime to Q.e Once modified the values
of the ui and vi , nothing changes from Chapter 5.

Case 5: Modelling a kernel element with multiplicity. Conversely, α(Pi − P∞ ) and

α(Pj − P∞ ) can also share the same point with multiplicities of identical sign, leading to multi-
plicities in the reduced divisor α(D). Similarly to what was done in the Case 5 of Section 5.4.1,
we can group the corresponding ui , uj , vi and vj in polynomials U and V such that U |V 2 − f
108 Chapter 7. Counting points on hyperelliptic curves with explicit RM

and deg V < deg U , and then look for ϕ = P (X) + Y Q(X) such that P + QV ≡ 0 mod U . Once
again, nothing changes apart from the definition of the ui ’s and vi ’s.

Case 3: Low weight after applying α. We kept this case for the end because it is not
a straightforward extension of the Case 3 appearing in Section 5.4.1. Until now, we assumed
that all the Pi ’s in the support of D were such that α(Pi − P∞ ) had weight g, i.e. dg (xi ) 6= 0.
We now want to model the case where D = w i=1 (Pi − P∞ ) such that each α(Pi − P∞ ) has
P

weight wi . In Chapter 5, this was done using a result from [28] giving a necessary and sufficient
condition for `(Pi − P∞ ) to be of weight wi . When α is an endomorphism other than scalar
multiplication, no such result holds a priori. In what follows, we solve this issue by designing
non-generic α-division polynomials Γα,t and ∆α,t such that α ((x, y) − P∞ ) has weight w if and
only if ∆α,w (x) = 0 and Γα,w−1 (x) 6= 0.

Combining all degeneracies. As in Section 5.4.2, we have to consider situations in which

several of the previous cases occur simultaneously. Note that while we wanted to compute
the whole `-torsion in Chapter 5, we now only need one kernel element per endomorphism αi
to determine χπ mod `. Therefore, after finding a non-zero solution to any of the subsequent
systems, one need not consider the others. Once again, we will not perform a complete analysis
as in Section 5.4.2 but rather detail when modifying the values of ui and vi is not sufficient. We
also update the analysis on the numbers and degrees of equations and variables. The aim of the
Section is to prove the following proposition.

Proposition 7.7. We can model the set of non-generic elements of J[α] as the solution set of
Oη (1) bihomogeneous polynomial systems each consisting of O(g 2 ) equations in Fq [X1 , . . . , Xg , Y1 ,
. . . , Yny ] such that ny = O(g 2 ) and the degrees in the Xi ’s and Yj ’s are respectively in Oη (`3/g )
and Oη (1).

Proof. We similarly encode each situation by a non-genericity tuple (w, λ, τ, ε, M ) in the sense
of Definition 5.8, and derive an associated polynomial system whose solution set corresponds to
elements D ∈ J[α] such that:
Pk
• the reduced divisor D of weight w has the form i=1 λi Pi with distinct Pi ’s,

• each λi α(Pi − P∞ ) has weight τi ,

• each εi is in {0, 1} and such that εi = 1 if and only if τi = λi = 1.

• the k × s matrix M represents the points shared by the λi α(Pi − P∞ ) as in Section 5.4.2,
with s ≤ gk.

We can follow the analysis of Section 5.4.2 to describe more explicitly the equations and their
degrees / number of variables, and remark that the only part that does not generalize readily is
the definition of non-generic α-division polynomials, as in the Case 3 above. Let us first fix this
issue.
When the weight ti of λi α(Pi − P∞ ) is strictly smaller than g, the usual coordinate system
given by the Mumford form is no longer available, due to the vanishing of the denominator
eg (xi ). We define an adequate coordinate system to describe non-generic elements of weight t.
Let us consider the variety

Vα,t = {(x, y) ∈ C | α ((x, y) − P∞ ) has weight t} .

7.3. Complexity analysis 109

We want to define polynomials ∆α,t and Γα,t such that a point is in Vα,w if and only if
∆α,w (x) = 0 and Γα,w−1 (x) 6= 0 iteratively. First, ∆α,g−1 = GCD(dg , eg ), so that the points
(x, y) of Vα,g−1 satisfy ∆α,g−1 (x, y) = 0. Assuming that for k < g we have already constructed a
squarefree polynomial ∆α,k vanishing on the abscissae of points in Vα,k , then one can compute
α ((x, y) − P∞ ) over Fp [x, y]/(∆α,k (x), y 2 − f (x)). By our recurrence hypothesis, the Mumford
form of the result is hu, vi, with u of degree k and v of degree k − 1. Let Γα,k−1 be the product of
LC(u) with the denominator of LC(v), then Vα,k is the set of points (x, y) such that ∆α,k (x) = 0
and Γα,k−1 (x) 6= 0. Furthermore, ∆α,k−1 = GCD(∆α,k , Γα,k−1 ) vanishes on the points of Vα,k−1 .
To avoid multiplicities, we replace ∆α,t (x) by the square-free polynomial whose roots are
exactly the roots of ∆α,t (x) that are not roots of Γα,t−1 (x) when it is necessary. Note that the
degrees of the ∆ and Γ are by construction bounded by deg ∆α,g−1 ≤ deg dg with deg dg itself
bounded by Oη (`1/g ). This way, we state an analogue of Definition 5.9 for non-generic α-division
polynomials:
Definition 7.8. The non-generic α-division polynomials uα,t and vα,t are the polynomials in X
with coefficients in Fp [x, y]/(∆α,t (x), y 2 − f (x)) such that
D E
α((x, y) − ∞) = uα,t (X), vα,t (X) ,

in weight-t Mumford representation: uα,t (X) is monic of degree t, v`,t (X) is of degree at most
t − 1 and they satisfy uα,t | v2α,t − f .
All the equations associated to a non-genericity tuple (w, λ, t, , M ) are merely identical to
those of Section 5.4.2 except that the di , ei and have different definitions and that ∆α,t replaces
˜ `,t so that Equation (Sys.3) now reads
∆
(
∆λi α,ti (xi ) = 0,
for all i in [1, k] such that ti < g. (Sys.3b)
Γλi α,ti −1 (xi ) 6= 0,

While turning the systems describing J[`] into systems describing J[α], we did not add any
variable, so that the study of Section 5.4.2 presented in Table 5.1 is still valid and we just recall
that the total number of variables is bounded by 4g 2 + g.
As for the number of equations and their respective degrees, the only change comes from the
fact that the coefficients of the ui and vi have degrees in the xi bounded by Oη (`3/g ) instead of
Oη (`3 ), and Table 5.2 becomes Table 7.1.
Table 7.1 shows that any system corresponding to a non-genericity tuple satisfies the degree
conditions of Proposition 7.7. As in the non-RM case, the number of such tuples is bounded by
3
g O(g ) and Proposition 7.7 is proved.

7.3 Complexity analysis

Now that we have modelled subsets of J[α] by polynomial systems whose size in terms of
equations, variables and degrees have been carefully bounded, we apply the geometric resolution
algorithm and bound its complexity using analogues of Proposition 5.3.

7.3.1 Solving the polynomial systems modelling J[α]

Proposition 7.9. For any ε > 0, there is a constant D such that for any endomorphism
α ∈ Z[η] of norm a multiple of ` > g coprime to the base field characteristic, there is a Monte
110 Chapter 7. Counting points on hyperelliptic curves with explicit RM

Equations reference Number of equations (and bound) deg1 deg2

Eq. and Ineq. Sys.1 2k ≤ 2g 2g + 1 0
InEq. Sys.2 k(k − 1)/2 ≤ g(g − 1)/2 1 0
Eq. and Ineq. Sys.3b ≤ 2g Oη (`3/g ) 0
InEq. Sys.4 ≤g Oη (`3/g ) 0
Pk Ps 4
Eq. Sys.5 i=1 j=1 |mij | ≤ g Oη (`3/g ) ≤g
InEq. Sys.6 ks ≤ g 3 Oη (`3/g ) ≤g
Eq. Sys.7 and Sys.8 ≤ k2 s ≤ g4 Oη (`3/g ) ≤g
InEq. Sys.9 ≤ s2 ≤ g 4 0 1
Pk 2
Eq. Sys.10 i=1 ti ≤ g Oη (`3/g ) ≤g
Eq. Sys.11 deg U ≤ g 2 0 O(g 3 )
Pk
Eq. Sys.12 i=1 deg uei ≤ g 2 Oη (`3/g ) O(g 2 )
Eq. Sys.13 deg U ≤ g 2 0 O(g 3 )

Table 7.1: Summary of the degrees of the equations in the polynomial system corresponding to
a normalized non-genericity tuple (w, λ, t, , M ).

Carlo algorithm which computes an Fqe -geometric resolution of the sub-variety of J[α] consisting
of α-generic α-torsion elements, where e = Oη (log `). The time and space complexities of this
algorithm are bounded by Oη (`D (log q)2+ε ) and it returns the correct result with probability at
least 5/6.

Proof. Let us consider the sub-variety S ⊂ J[α] consisting of α-generic elements, and I the
corresponding ideal. More precisely, we see I as the ideal of a sub-scheme of the scheme J[α],
itself subscheme of J[deg α], which is the kernel of a finite and étale map because deg α is a
small multiple of ` and is hence coprime to the characteristic p thanks to our assumptions on
the size of p in the statement of Theorem 7.1.
Therefore, I is 0-dimensional and radical. Since all the elements in S have the same weight g
we can use the Mumford coordinates hu(X), v(X)i with deg u = g and deg v < g − 1 as a
local system of coordinates to represent them. But the polynomial system that we have built
is with the (xi , yi ) coordinates, that is, it generates the ideal I unsym obtained by adjoining to
the equations defining I the 2g equations coming from u(X) = (X − xi ) and yi = v(xi ).
Q

Then we have deg I unsym = g! deg I. By the α-genericity condition, all the fibers in the variety
have exactly g! distinct points corresponding to permuting the (xi , yi ) which are all distinct.
Therefore the radicality of I implies the radicality of I unsym and we can apply a modified version
of Proposition 5.3 to our polynomial system.
Indeed, by Proposition 7.6 we now have a function h and a constant C such that dx ≤
h(g)`C/g instead of h(g)`C . This propagates in the proof of Proposition 5.3, and since the power
of ` only comes from the bound on dx , we can also replace ` by `1/g in the final result, so that we
can compute an Fqe -geometric resolution of S in time and space bounded by Oη (`D (log q)2+ε ),
with e = Oη (log `).

Following the same proof but invoking Proposition 7.7 instead of Proposition 7.6, the same
complexity bound holds for solving the polynomial system associated to any non-genericity tuple.
Even if a non-zero α-torsion element is only found after solving all the systems associated to
non-genericity tuples, the cost for computing ψ mod ` is only multiplied by a factor in Oη (1).
We have proven that there exists a constant c such that for any prime ` satisfying conditions
(C1) to (C4), computing χπ mod ` is achieved within Oη (`c ) field operations. Taking into account
7.3. Complexity analysis 111

the size of the largest ` to consider and the cost of field operations, the overall complexity of our
point-counting algorithm is in Oη ((log q)c+2 ). The bottleneck is computing geometric resolutions
of polynomial systems which is quadratic in their respective multihomogeneous Bézout bounds,
up to a factor in Oη (1). Still neglecting factors in Oη (1), the multihomogeneous Bézout bound
itself boils down to Oη (degg1 ) by Definition 2.44. As shown in Table 7.1, deg1 = Oη (`3/g ) so we
deduce that c = 6 and get an overall complexity bound in O e η (log8 q).
Note that our bound on deg1 is pessimistic because we used the proven cubic bound for the
degrees of Cantor’s division polynomials while we expect them to be actually quadratic. Under
this assumption, deg1 is reduced to Oη (`2/g ) and the overall complexity would therefore be in
Oe η (log6 q) for any g. Apart from the part depending on g, this conjectural result is identical to
what we proved for genus 3. In the next section, we push the analysis forward by investigating
the dependency on g.

7.3.2 Dependency on g of the complexity

The goal of this section is to assess the potential of our algorithm to achieve a polynomial-time
complexity both in g and log q on some family of curves. To this end, we review our complexity
analysis with additional attention given to the factors that previously vanished in the Oη .

Dependency on g of the largest ` Let us first come back to the constant Cg of Section 7.1.2.
We have seen that the only non-polynomial dependency on g came from the matrix norm when
inverting the linear change of variables ψk = g−1 i
P
i=0 ai ηk , which is described by the Vandermonde
matrix of the g conjugates of η, denoted by ηk for k ∈ {1, . . . , g}. Let B the inverse of this
matrix, then we have P j−1 η · · · η
1≤k1 <···<kg−j <g (−1) k1 kg−j
k1 ,...,kg−j 6=i
Bij = .
− ηi )
Q
ηi k6=i (ηk

Let E = maxk (|η1 |, . . . , |ηk |), e = 1/ mink (|η1 |, . . . , |ηk |), and D = maxi6=j |ηi − ηj |−1 , then we

can bound the absolute value of any entry of B very roughly either by ge(2ED)g or by ge if
2ED ≤ 1, and the matrix-norm of B is bounded by g times this previous bound. Note that the
factor ∆ is also a nuisance but it is bounded by the discriminant of Z[η]. This discriminant is
in turn bounded by maxi6=j (|ηi − ηj |)2g . Thus, the constant Cg can be bounded by g 2 cg , where
c has a polynomial dependency on η and its conjugates.
Q √
By the prime number theorem, the set L of primes such that `∈L ` > 2Cg q is such that
the number and size of primes in L is in O(g) e log q/ log log q. As we already mentioned, the
primes to consider must satisfy the conditions (C1) to (C4) and that may cause them to be
larger by a factor depending exponentially on g a priori. Since the complexity of computing
χπ mod ` is polynomial in `, this implies that the overall complexity depends exponentially on
g in general.
However, a curve in the family Cn,t introduced in Section 7.1.1 has RM by the real subfield
of Q(ζn ), for which we know that the proportion of split primes is 2/(n − 1) = 1/g. Therefore,
this first obstacle due to the size of primes to consider can be overcome provided that we further
strengthen the assumptions on the RM-curves we consider.

Finding small elements in lattices This time, the exhaustiven search is no longer
o sufficient
g
for our need because of the factor 2 in the size of the ball v | ||v||∞ ≤ ∆ `1/g 1/g . Unfortu-
nately, the current best known algorithms for finding short vectors in time subexponential in
112 Chapter 7. Counting points on hyperelliptic curves with explicit RM

the dimension of the lattice have a drawback that makes them unusable in our point-counting
algorithm. Indeed, although they run faster than the naive approach, they do not necessarily
output the shortest non-zero vector in the lattice, but an approximation that may be greater
by a factor which is also subexponential in the dimension. The size of the short vector plays a
prominent role in the complexity analysis of our point-counting algorithm as it gives a bound on
the degrees of the equations modelling J[α]. Even if we find an α whose coordinates are in g`1/g
instead of c`1/g , the factor g will cause a factor g g in the multihomogeneous Bézout bound, and
hence in the final complexity of solving the polynomial systems.
Although finding short generators of ideals in number fields is believed to be hard in general,
we may still expect to further restrict the RM curves we consider so as to fall in a case for which
the complexity of such task becomes affordable. Examples are given in [13], where a classical
algorithm is shown to compute short generators √ of principal
√ ideals in particular number fields
called multiquadratics, i.e. fields of the form Q( d1 , . . . , dn ). While we acknowledge that it
is quite speculative to hope for families of curves of arbitrary high genus with RM by a Z[η]
satisfying all the previous hypotheses, we do not linger on this because the next point is much
more of a concern anyway.

Solving polynomial systems Using the strategy of Section 7.2, the complexity is polynomial
2
in the multihomogeneous Bézout bound, itself including a combinatorial factor in g g . Indeed,
although the ideals of α-torsion have degree `2 independent of g, this is not true for the number
of variables involved in our modelling, which is at least g 2 in the generic case. Worse than that,
the size of the polynomial systems modelling the set of generic α-torsion elements is already
exponential in g. Indeed, following the proof of Lemma 2.50, one sees that the number of
nx +dx

monomials has a factor nx and our modelling is such that nx = g and dx ≥ g`2/g so that
nx +dx

nx ≥ (nx + dx )nx /nnx x is bounded below by g g .
Thus, there is no hope of turning our algorithm into something subexponential in g in its
current state. Possible workarounds could be looking for easier instances in which we could
model the α-torsion by even smaller polynomial systems, or cases for which there are simpler
ways of obtaining a generic α-torsion divisor than the one we used.
 Conclusion

In this thesis, we focus on point-counting on hyperelliptic curves over finite fields using methods
derived from Schoof and Pila’s algorithms. We have studied the asymptotic complexity of this
task for curves of arbitrary genus defined over a sufficiently large field. In particular, the power
of log q in the complexity has been reduced from O(g 2 log g) to O(g) in Chapter 5. For families
of curves equipped with an explict RM, we have further reduced this power to a constant in
Chapter 7, and proved that our algorithm computes the zeta function of genus-g hyperelliptic
curves with explicit RM in time bounded by O e η (log8 q). Conjecturally, we actually expect this
6
complexity to be in O e η (log q).

Instanciating our general methods in small genus

It is natural to wonder whether the algorithm we described in Chapter 5 to establish those

complexity bounds, or rather their instantiations for curves of small genus, are competitive with
the previously existing extensions of Schoof’s algorithm in terms of complexity. In genus 2,
8
our general method cannot improve on the complexity in O(loge q) of the Gaudry-Harley-Schost
 
algorithm based on resultants, which can actually be further reduced to O (log q)8−2/ω+ε using
the algorithm of Villard [143] for bivariate resultants. Indeed, the `-torsion ideals involved have
degrees in O(`4 ), so that our algorithm based on geometric resolution requires a number of field
10
operations at least quadratic in that degree, i.e. an overall cost of at least O(log
e q) because of
the number and sizes of primes ` to consider.
In genus 3, no previous instantiation of the Schoof-Pila algorithm had been presented and
the complexity of a potential extension was subject to speculation. Since the `-torsion ide-
als have degrees in O(`6 ), we expect our general algorithm to have a complexity at least in
14
O(log
e q). Extending the resultant-based elimination scheme of Gaudry-Harley-Schost, we ob-
14
tained a proven complexity in O(log
e q) in Chapter 6. In fact, using the algorithm of Villard for
computing
 bivariate resultants,
 the complexity of the resultant-based approach can be decreased
to O (log q) 14−4/ω+ε , which is also less than quadratic in the degree of the `-torsion.
Still in genus 3 but for hyperelliptic curves with explicit RM, we have to solve systems
of much smaller degrees, and we turned this into a point-counting algorithm of complexity
6
O(log
e q). Setting g = 3 in the general counterpart to this algorithm designed in Chapter 7, we
achieve a similar complexity because for g ≤ 3 the conjectural result leading to the Oe η (log6 q) is
actually proven. However, using Villard’s algorithm for bivariate resultants, the resultant-based
algorithm of Chapter 6 reaches a complexity in O((log q)6−4/(3ω)+ε ) which is once again better
than the general approach. We sum up all these results in Table 2.

113
114 Conclusion

Table 2: Asymptotic complexities for counting points on hyperelliptic curves of genus ≤ 3

Approach g=2 g=3 g = 2 with RM g = 3 with RM

10 14 6 6
Chapters 5 and 7 O(log
e q) O(log
e q) O(log
e q) O(log
e q)
8 14 5 6
Chapters 3 and 6 O(log
e q) O(log
e q) O(log
e q) O(log
e q)
Using [143] O((log q)8−2/ω+ε ) O((log q)14−4/ω+ε ) O((log q)5−1/ω+ε ) O((log q)6−4/(3ω)+ε )

Practical experiments
In practice, we expect our general methods to be no match for the tailor-made algorithms in
genus ≤ 3, not only because their complexities are lower, but also because the general approaches
hide constants that we expect to be much larger. However, a comparison based on practical
experiments in full generality is unrealistic in genus ≥ 3 because of the prohibitive complexities
of both the algorithms of Chapters 5 and 7.
For practical experiments in genus 3, we considered the easier case of curves with explicit real
multiplication, an approach that had previously been studied with benefit in genus 2. We were
able to successfully count points on a genus-3 curve defined over F264 −59 . This is comparable in
size with previous record computations due to Sutherland using generic group methods which
also take advantage of particularities of the input curves, although such peculiar curves are
more frequent than curves with explicit RM. In our practical experiments, we used a trivariate
elimination scheme except that we computed Gröbner bases instead of trivariate resultants.
The complexity estimate in O e η (log6 q) conjectured in Chapter 7 could give hope of pushing
practical experiments to higher genus, since the exponent of log q is independent of g. However,
considering the RM families we presented and the conditions on primes `, the smallest example
available in genus larger than 3 is the computation of the 23-torsion of a hyperelliptic curve
of genus 5. Even over a relatively small finite field, this is unrealistic because the systems to
solve would have 5 variables with “large” degrees (estimated to be at least 10) and at least 25
variables with degree 1.

Prospective
A natural question that applies to all of our contributions is the possibility of extending our
complexity bounds to non-hyperelliptic curves. Even if the Mumford representation allows for
a much more straightforward representation of elements and simpler conditions to express the
nullity of an element, this is not an absolute necessity. The most important result is that the
degree of the `-torsion ideal is still `2g in any Jacobian of a genus-g curve. Provided that we
can model this ideal by a polynomial system with a number of variables that depends only
on its dimension g and such that “only” O(g c ) of them have degrees actually depending on
`, then
 the geometric
 resolution algorithm yields a point-counting algorithm running in time
O(g c)
Og (log q) . Controlling the constant c and giving an explicit bound would already improve
the result of Adleman and Huang [3], but we expect that it should be possible to prove that
c = 1 as we did in the hyperelliptic case, at least for Jacobians of plane curves.
In Chapter 5, we perform a tedious analysis of how to handle non-generic elements in the
 115

torsion subgroups. It is quite unsatisfactory that such amount of work is performed for cases
which are supposed not to happen, or with an incredibly low probability. Actually, while we
consider many cases, we do not even prove that they happen. Therefore, one could wonder
whether all those non-genericities are possible. In Chapter 7, non-genericities are even less likely
to become a nuisance since it is sufficient to have only one generic element in the kernel of our
endomorphisms. Even better, one could try to completely remove the non-genericity analysis
by proving for instance that given a curve, the proportion of primes ` for which non-genericity
occurs is finite or sufficiently small. Conversely, a skeptical reader could attempt to create
pathological curves such that avoiding all the “bad” primes ` would entail considering primes
sufficiently large to hamper our complexity result. Note that because of our bounds, this would
require finding a family of curves such that the largest required prime ` grows faster than any
power of log q.
The question of finding a classical point-counting algorithm running in time polynomial in
both g and log q being open, we wonder whether the approach of Chapter 7 has the potential
for providing a small yet non-trivial family of curves for which such an algorithm exists. The
first reason why the algorithm presented in Chapter 7 is exponential in g is that the multiho-
2
mogeneous Bézout bound has a combinatorial factor in O(g g ). Indeed, even though we manage
to decrease the degrees of the equations by splitting the `-torsion into a direct sum of kernels
of endomorphisms of degree `2 , our systems still have O(g 2 ) variables. We have reviewed the
other sources of factors exponential in g, and remarked that the polynomial systems appearing
in the modelling come both in number and size exponential in g. Therefore, our approach needs
further insight before turning into an algorithm running in time subexponential in g, even on a
particular subset of curves.
116 Conclusion
 Bibliography

[1] Simon Abelard, Pierrick Gaudry, and Pierre-Jean Spaenlehauer. Improved complexity
bounds for counting points on hyperelliptic curves, 2017. To appear in Foundations of
Computational Mathematics, ArXiv preprint 1710.03448.

[2] Simon Abelard, Pierrick Gaudry, and Pierre-Jean Spaenlehauer. Counting points on
genus-3 hyperelliptic curves with explicit real multiplication, 2018. To appear in the
Proceedings of the ANTS-XIII Conference (Thirteenth Algorithmic Number Theory Sym-
posium), ArXiv preprint 1806.05834.

[3] Leonard M. Adleman and Ming-Deh Huang. Counting points on curves and Abelian
varieties over finite fields. Journal of Symbolic Computation, 32(3):171–189, 2001.

[4] Leonard M. Adleman and Ming-Deh A. Huang. Primality testing and Abelian varieties
over finite fields. Springer, 2006.

[5] François Apéry and Jean-Pierre Jouanolou. Élimination : le cas d’une variable. Hermann,
Collection Méthodes, 2006.

[6] A. Oliver L. Atkin and François Morain. Finding suitable curves for the elliptic curve
method of factorization. Mathematics of Computation, 60(201):399–405, 1993.

[7] A. Oliver L. Atkin and François Morain. Elliptic curves and primality proving. Mathe-
matics of Computation, 61(203):29–68, 1993.

[8] Sean Ballentine, Aurore Guillevic, Elisa Lorenzo García, Chloe Martindale, Maike
Massierer, Benjamin Smith, and Jaap Top. Isogenies for point counting on genus two
hyperelliptic curves with maximal real multiplication. In Algebraic Geometry for Coding
Theory and Cryptography, pages 63–94. Springer, 2017.

[9] Stéphane Ballet, Julia Pieltant, Matthieu Rambaud, and Jeroen Sijsling. On some bounds
for symmetric tensor rank of multiplication in finite fields. Contemporary Mathematics,
AMS, 686:93–121, 2017.

[10] Razvan Barbulescu, Joppe W. Bos, Cyril Bouvier, Thorsten Kleinjung, and Peter L. Mont-
gomery. Finding ECM-friendly curves through a study of Galois properties. In ANTS X,
volume 1 of The open book series, pages 63–86, 2012.

[11] Magali Bardet. Étude des systèmes algébriques surdéterminés. Applications aux codes
correcteurs et à la cryptographie. PhD thesis, Université Pierre et Marie Curie-Paris VI,
2004.

117
118 Bibliography

[12] Magali Bardet, Jean-Charles Faugère, and Bruno Salvy. On the complexity of the F5
Gröbner basis algorithm. Journal of Symbolic Computation, 70:49–70, 2015.

[13] Jens Bauch, Daniel J. Bernstein, Henry de Valence, Tanja Lange, and Christine Van Vre-
dendaal. Short generators without quantum computers: the case of multiquadratics. In
EUROCRYPT 2017, volume 10210 of LNCS, pages 27–59. Springer, 2017.

[14] Eberhard Becker, Teo Mora, Maria Grazia Marinari, and Carlo Traverso. The shape of
the shape lemma. In Proceedings of ISSAC 1994, pages 129–133. ACM, 1994.

[15] Elwyn R. Berlekamp. Factoring polynomials over large finite fields. Mathematics of Com-
putation, 24(111):713–735, 1970.

[16] Daniel Bernstein, Peter Birkner, Tanja Lange, and Christiane Peters. ECM using Edwards
curves. Mathematics of Computation, 82(282):1139–1179, 2013.

[17] Daniel J. Bernstein. Curve25519: new Diffie–Hellman speed records. In PKC 2006, volume
3958 of LNCS, pages 207–228. Springer, 2006.

[18] Daniel J. Bernstein, Chitchanok Chuengsatiansup, Tanja Lange, and Peter Schwabe. Kum-
mer strikes back: new DH speed records. In ASIACRYPT 2014, volume 8873 of LNCS,
pages 317–337. Springer, 2014.

[19] Christina Birkenhake and Herbert Lange. Complex Abelian varieties. Springer-Verlag,
2004.

[20] G. Bisson, R. Cosset, D. Robert, et al. AVIsogenies (abelian varieties and isogenies).
Magma package for explicit isogenies between abelian varieties, 2010.

[21] Alin Bostan, Frédéric Chyzak, Marc Giusti, Romain Lebreton, Grégoire Lecerf, Bruno
Salvy, and Éric Schost. Algorithmes efficaces en calcul formel. Published by the authors,
2017.

[22] Alin Bostan, Grégoire Lecerf, Bruno Salvy, Éric Schost, and Bernd Wiebelt. Complexity
issues in bivariate polynomial factorization. In Proceedings of ISSAC 2004, pages 42–49.
ACM, 2004.

[23] Ivan Boyer. Variétés abéliennes et jacobiennes de courbes hyperelliptiques, en particulier

à multiplication réelle ou complexe. PhD thesis, Paris 7, 2014.

[24] Richard P. Brent and Paul Zimmermann. Modern computer arithmetic. Cambridge Uni-
versity Press, 2010.

[25] Antonio Cafure and Guillermo Matera. Fast computation of a rational point of a variety
over a finite field. Mathematics of Computation, 75(256):2049–2085, 2006.

[26] Antonio Cafure and Guillermo Matera. An effective Bertini theorem and the number of
rational points of a normal complete intersection over a finite field. Acta Arithmetica,
130(1):19–35, 2007.

[27] David G. Cantor. Computing in the Jacobian of a hyperelliptic curve. Mathematics of

computation, 48(177):95–101, 1987.
 119

[28] David G. Cantor. On the analogue of the division polynomials for hyperelliptic curves.
Journal fur die reine und angewandte Mathematik, 447:91–146, 1994.

[29] Wouter Castryck, Jan Denef, and Frederik Vercauteren. Computing zeta functions of
nondegenerate curves. International Mathematics Research Papers, Vol. 2006, 2006.

[30] Wouter Castryck, Hendrik Hubrechts, and Frederik Vercauteren. Computing zeta func-
tions in families of Ca,b curves using deformation. In ANTS 2008, volume 5011 of LNCS,
pages 296–311. Springer, 2008.

[31] David Volfovich Chudnovsky and Gregory Volfovich Chudnovsky. Algebraic complexities
and algebraic curves over finite fields. Journal of Complexity, 4(4):285–316, 1988.

[32] Laurent Clozel, Michael Harris, and Richard Taylor. Automorphy for some `-adic lifts of
automorphic mod ` Galois representations. Publications mathématiques, 108(1):1, 2008.

[33] Henri Cohen. A course in computational algebraic number theory. Springer, 1993.

[34] Henri Cohen, Gerhard Frey, Roberto Avanzi, Christophe Doche, Tanja Lange, Kim
Nguyen, and Frederik Vercauteren. Handbook of elliptic and hyperelliptic curve cryp-
tography. CRC press, 2005.

[35] Stéphane Collart, Michael Kalkbrener, and Daniel Mall. Converting bases with the Gröb-
ner walk. Journal of Symbolic Computation, 24(3-4):465–469, 1997.

[36] Nicolas Courtois, Alexander Klimov, Jacques Patarin, and Adi Shamir. Efficient algo-
rithms for solving overdefined systems of multivariate polynomial equations. In EURO-
CRYPT 2000, volume 1807 of LNCS, pages 392–407. Springer, 2000.

[37] Jean-Marc Couveignes and Tony Ezome. Computing functions on Jacobians and their
quotients. LMS Journal of Computation and Mathematics, 18(1):555–577, 2015.

[38] David Cox, John Little, and Donal O’Shea. Ideals, varieties, and algorithms, volume 3.
Springer, 2007.

[39] Ronald Cramer, Léo Ducas, Chris Peikert, and Oded Regev. Recovering short generators
of principal ideals in cyclotomic rings. In EUROCRYPT 2016, volume 9666 of LNCS,
pages 559–585. Springer, 2016.

[40] Jean Della Dora, Claire Dicrescenzo, and Dominique Duval. About a new method for
computing in algebraic number fields. In European Conference on Computer Algebra,
pages 289–290. Springer, 1985.

[41] Jan Denef and Frederik Vercauteren. An extension of Kedlaya’s algorithm to hyperelliptic
curves in characteristic 2. Journal of Cryptology, 19(1):1–25, 2006.

[42] Noam D. Elkies. Elliptic and modular curves over finite fields and related computational
issues. In Computational Perspectives on Number Theory, pages 21–76. AMS/International
Press, 1998. Proceedings of a Conference in Honor of A.O.L. Atkin.

[43] Jordan S. Ellenberg. Endomorphism algebras of Jacobians. Advances in Mathematics,

162:243–271, 2001.
120 Bibliography

[44] Andreas Enge and Emmanuel Thomé. Computing class polynomials for abelian surfaces.
Experimental Mathematics, 23(2):129–145, 2014.

[45] Jean-Charles Faugère. A new efficient algorithm for computing Gröbner bases (F4). Jour-
nal of pure and applied algebra, 139(1-3):61–88, 1999.

[46] Jean-Charles Faugère. A new efficient algorithm for computing Gröbner bases without
reduction to zero (F5). Proceedings of ISSAC 2002, 2002.

[47] Jean-Charles Faugère, Patrizia Gianni, Daniel Lazard, and Teo Mora. Efficient compu-
tation of zero-dimensional Gröbner bases by change of ordering. Journal of Symbolic
Computation, 16(4):329–344, 1993.

[48] Jean-Charles Faugère, David Lubicz, and Damien Robert. Computing modular correspon-
dences for abelian varieties. Journal of Algebra, 343(1):248–277, 2011.

[49] Francesc Fité, Kiran S. Kedlaya, Víctor Rotger, and Andrew V. Sutherland. Sato–Tate
distributions and Galois endomorphism modules in genus 2. Compositio Mathematica,
148(5):1390–1442, 2012.

[50] Gerhard Frey and Michael Müller. Arithmetic of modular curves and applications. In
B. Heinrich Matzat, Gert-Martin Greuel, and Gerhard Hiss, editors, Algorithmic Algebra
and Number Theory, pages 11–48. Springer Verlag, 1999.

[51] Matsusaburô Fujiwara. Über die obere schranke des absoluten betrages der wurzeln einer
algebraischen gleichung. Tohoku Mathematical Journal, First Series, 10:167–171, 1916.

[52] Steven Galbraith and Raminder S. Ruprai. An improvement to the Gaudry-Schost algo-
rithm for multidimensional discrete logarithm problems. In IMA International Conference
on Cryptography and Coding, volume 5921 of LNCS, pages 368–382. Springer, 2009.

[53] Steven D. Galbraith and Pierrick Gaudry. Recent progress on the elliptic curve discrete
logarithm problem. Designs, Codes and Cryptography, 78(1):51–72, 2016.

[54] Joachim von zur Gathen and Jürgen Gerhard. Modern computer algebra. Cambridge
university press, 2013. Third edition.

[55] Pierrick Gaudry. Fast genus 2 arithmetic based on theta functions. Journal of Mathemat-
ical Cryptology JMC, 1(3):243–265, 2007.

[56] Pierrick Gaudry. Algorithmes de comptage de points d’une courbe définie sur un corps
fini, 2013.

[57] Pierrick Gaudry and Robert Harley. Counting points on hyperelliptic curves over finite
fields. In ANTS 2000, volume 1838 of LNCS, pages 313–332. Springer, 2000.

[58] Pierrick Gaudry, Florian Hess, and Nigel P. Smart. Constructive and destructive facets of
Weil descent on elliptic curves. Journal of Cryptology, 15(1):19–46, 2002.

[59] Pierrick Gaudry, David R. Kohel, and Benjamin A. Smith. Counting points on genus
2 curves with real multiplication. In ASIACRYPT 2011, volume 7073 of LNCS, pages
504–519. Springer, 2011.
 121

[60] Pierrick Gaudry and Éric Schost. Construction of secure random curves of genus 2 over
prime fields. In EUROCRYPT 2004, volume 3027 of LNCS, pages 239–256. Springer,
2004.

[61] Pierrick Gaudry and Éric Schost. A low-memory parallel version of Matsuo, Chao and
Tsujii’s algorithm. In ANTS-VI, volume 3076 of LNCS, pages 208–222. Springer Verlag,
2004.

[62] Pierrick Gaudry and Éric Schost. Genus 2 point counting over prime fields. Journal of
Symbolic Computation, 47(4):368–400, 2012.

[63] Gerard van der Geer, Everett W. Howe, Kristin E. Lauter, and Christophe Ritzenthaler.
Tables of curves with many points, 2009.

[64] Marc Giusti, Grégoire Lecerf, and Bruno Salvy. A Gröbner free alternative for polynomial
system solving. Journal of complexity, 17(1):154–211, 2001.

[65] Shafi Goldwasser and Joe Kilian. Almost all primes can be quickly certified. In Proceedings
of the eighteenth annual ACM symposium on Theory of computing, pages 316–329. ACM,
1986.

[66] Valery Denisovich Goppa. Algebraico-geometric codes. Izvestiya: Mathematics, 21(1):75–

91, 1983.

[67] Michael Harris, Nick Shepherd-Barron, and Richard Taylor. A family of Calabi-Yau vari-
eties and potential automorphy. Annals of Mathematics, pages 779–813, 2010.

[68] Robin Hartshorne. Algebraic geometry. Springer, 1977.

[69] David Harvey. Computing zeta functions of arithmetic schemes. Proceedings of the London
Mathematical Society, 111(6):1379–1401, 2015.

[70] David Harvey and Andrew V. Sutherland. Computing Hasse–Witt matrices of hyperel-
liptic curves in average polynomial time. LMS Journal of Computation and Mathematics,
17(A):257–273, 2014.

[71] David Harvey and Andrew V. Sutherland. Computing Hasse-Witt matrices of hyperelliptic
curves in average polynomial time II. Contemporary Mathematics, 663:127–148, 2016.

[72] Joos Heintz. Definability and fast quantifier elimination in algebraically closed fields.
Theoretical Computer Science, 24(3):239–277, 1983.

[73] Florian Hess. Computing Riemann–Roch spaces in algebraic function fields and related
topics. Journal of Symbolic Computation, 33(4):425–445, 2002.

[74] Alston S. Householder. The Padé table, the Frobenius identities, and the qd algorithm.
Linear Algebra and its applications, 4(2):161–174, 1971.

[75] Ming-Deh Huang and Doug Ierardi. Counting points on curves over finite fields. Journal
of Symbolic Computation, 25(1):1–21, 1998.

[76] Peter Kaiser. Greatest numbers certified with Primo. http://primes.utm.edu/primes/

page.php?id=123996.
122 Bibliography

[77] Wolfgang Kampkötter. Explizite gleichungen für Jacobische varietäten hyperelliptischer

kurven. Universität Essen. Institut für Experimentelle Mathematik, 1991.
[78] Nicholas M. Katz and Peter Sarnak. Random matrices, Frobenius eigenvalues, and mon-
odromy, volume 45. AMS, 1999.
[79] Motoko Qiu Kawakita. Certain sextics with many rational points. Advances in Mathe-
matics of Communications, 11(2), 2017.
[80] Kiran S. Kedlaya. Counting points on hyperelliptic curves using Monsky-Washnitzer co-
homology. Journal of the Ramanujan Mathematical Society, 16(4):323–338, 2001.
[81] Kiran S. Kedlaya. Quantum computation of zeta functions of curves. computational
complexity, 15(1):1–19, 2006.
[82] Kiran S. Kedlaya and Andrew V. Sutherland. Hyperelliptic curves, L-polynomials, and
random matrices. Contemporary Mathematics, 14:119, 2009.
[83] Kamal Khuri-Makdisi. Linear algebra algorithms for divisors on an algebraic curve. Math-
ematics of Computation, 73(245):333–357, 2004.
[84] Steven L. Kleiman. Bertini and his two fundamental theorems. ArXiv e-print alg-
geom/9704018v1, 1997.
[85] Neal Koblitz. Hyperelliptic cryptosystems. Journal of cryptology, 1(3):139–150, 1989.
[86] Neal Koblitz. CM-curves with good cryptographic properties. In Annual International
Cryptology Conference, pages 279–287. Springer, 1991.
[87] David R. Kohel and Benjamin A. Smith. Efficiently computable endomorphisms for hy-
perelliptic curves. In ANTS VII, volume 4076 of LNCS, pages 495–509. Springer Verlag,
2006.
[88] Junichi Kuroki, Masaki Gonda, Kazuto Matsuo, Jinhui Chao, and Shigeo Tsujii. Fast
genus three hyperelliptic curve cryptosystems. In The 2002 Symposium on Cryptography
and Information Security, Japan—SCIS, 2002.
[89] Yagati N. Lakshman and Daniel Lazard. On the complexity of zero-dimensional algebraic
systems. In Effective methods in algebraic geometry, pages 217–225. Springer, 1991.
[90] Tanja Lange. Formulae for arithmetic on genus 2 hyperelliptic curves. Applicable Algebra
in Engineering, Communication and Computing, 15(5):295–328, 2005.
[91] Daniel Lazard. Gröbner bases, Gaussian elimination and resolution of systems of algebraic
equations. In European Conference on Computer Algebra, pages 146–156. Springer, 1983.
[92] Daniel Lazard. Solving systems of algebraic equations. ACM SIGSAM Bulletin, 35(3):11–
37, 2001.
[93] Hendrik W. Lenstra Jr. Factoring integers with elliptic curves. Annals of mathematics,
pages 649–673, 1987.
[94] Reynald Lercier and David Lubicz. Counting points on elliptic curves over finite fields
of small characteristic in quasi quadratic time. In EUROCRYPT 2003, volume 2656 of
LNCS, pages 360–373. Springer, 2003.
 123

[95] Reynald Lercier and David Lubicz. A quasi quadratic time algorithm for hyperelliptic
curve point counting. The Ramanujan Journal, 12(3):399–423, 2006.

[96] David Lubicz and Damien Robert. Arithmetic on Abelian and Kummer varieties. Finite
Fields and Their Applications, 39:130–158, 2016.

[97] F. S. Macaulay. Some formulae in elimination. Proceedings of the London Mathematical

Society, 1(1):3–27, 1902.

[98] Chloe Martindale. Isogeny graphs, modular polynomials, and applications. PhD thesis,
PhD thesis, Universiteit Leiden, 2017. in preparation, 2018.

[99] Kazuto Matsuo, Jinhui Chao, and Shigeo Tsujii. An improved baby step giant step algo-
rithm for point counting of hyperelliptic curves over finite fields. In ANTS 2002, volume
2369 of LNCS, pages 461–474. Springer, 2002.

[100] Ernst W. Mayr and Albert R. Meyer. The complexity of the word problems for com-
mutative semigroups and polynomial ideals. Advances in mathematics, 46(3):305–329,
1982.

[101] Jean-François Mestre. Familles de courbes hyperelliptiques à multiplications réelles. In

Arithmetic algebraic geometry, pages 193–208. Springer, 1991.

[102] Jean-François Mestre. Lettre adressée à Gaudry et Harley. Available on http://www. math.
jussieu. fr/˜ mestre, 2000.

[103] Enea Milio. Calcul de polynômes modulaires en dimension 2. PhD thesis, Université de
Bordeaux, 2015.

[104] Enea Milio. Computing isogenies between Jacobian of curves of genus 2 and 3. working
paper or preprint, September 2017.

[105] Victor S. Miller. Use of elliptic curves in cryptography. In CRYPTO 1985, volume 218 of
LNCS, pages 417–426. Springer, 1985.

[106] James S. Milne. Jacobian varieties. In Arithmetic geometry, pages 167–212. Springer,
1986.

[107] Shinji Miura. Algebraic geometric codes on certain plane curves. Electronics and Com-
munications in Japan (Part III: Fundamental Electronic Science), 76(12):1–13, 1993.

[108] Peter L. Montgomery. Speeding the Pollard and elliptic curve methods of factorization.
Mathematics of computation, 48(177):243–264, 1987.

[109] François Morain. Primality proving using elliptic curves: an update. In ANTS 1998,
volume 1423 of LNCS, pages 111–127. Springer, 1998.

[110] Alexander Morgan and Andrew Sommese. A homotopy for solving general polynomial
systems that respects m-homogeneous structures. Applied Mathematics and Computation,
24(2):101–113, 1987.

[111] David Mumford, Chidambaran Padmanabhan Ramanujam, and Jurij Ivanovič Manin.
Abelian varieties, volume 108. Oxford university press Oxford, 1974.
124 Bibliography

[112] Vassiliy Ilyich Nechaev. Complexity of a determinate algorithm for the discrete logarithm.
Mathematical Notes, 55(2):165–172, 1994.

[113] Committee on National Security Systems. Use of public standards for the se-
cure sharing of information among national security systems, advisory memorandum.
https://cryptome.org/2015/08/CNSS_Advisory_Memo_02-15.pdf, 2015.

[114] Jonathan Pila. Frobenius maps of Abelian varieties and finding roots of unity in finite
fields. Mathematics of Computation, 55(192):745–763, 1990.

[115] Jonathan Pila. Counting points on curves over families in polynomial time. arXiv preprint
math/0504570, 2005.

[116] Stephen Pohlig and Martin Hellman. An improved algorithm for computing logarithms
over GF(p) and its cryptographic significance (corresp.). IEEE Transactions on informa-
tion Theory, 24(1):106–110, 1978.

[117] Bjorn Poonen. Using zeta functions to factor polynomials over finite fields. arXiv preprint
arXiv:1710.00970, 2017.

[118] Matthieu Rambaud. Finding optimal Chudnovsky-Chudnovsky multiplication algorithms.

In International Workshop on the Arithmetic of Finite Fields, pages 45–60. Springer, 2014.

[119] Joost Renes, Peter Schwabe, Benjamin Smith, and Lejla Batina. µKummer: Efficient
hyperelliptic signatures and key exchange on microcontrollers. In CHES 2016, volume
9813 of LNCS, pages 301–320. Springer, 2016.

[120] Joost Renes and Benjamin Smith. qDSA: Small and secure digital signatures with curve-
based Diffie–Hellman key pairs. In ASIACRYPT 2017, volume 10625 of LNCS, pages
273–302. Springer, 2017.

[121] Christophe Ritzenthaler. Point counting on genus 3 non hyperelliptic curves. In ANTS
2004, volume 3076 of LNCS, pages 379–394. Springer, 2004.

[122] Damien Robert. Fonctions thêta et applications à la cryptographie. PhD thesis, Université
Henri Poincaré-Nancy I, 2010.

[123] Mohab Safey El Din and Éric Schost. A nearly optimal algorithm for deciding connectivity
queries in smooth and bounded real algebraic sets. Journal of the ACM, 63(6):1– 48, 2017.

[124] Takakazu Satoh. The canonical lift of an ordinary elliptic curve over a finite field and its
point counting. Journal of the Ramanujan Mathematical Society, 15(4):247–270, 2000.

[125] Takakazu Satoh. On p-adic point counting algorithms for elliptic curves over finite fields.
In International Algorithmic Number Theory Symposium, volume 2369 of LNCS, pages
43–66. Springer, 2002.

[126] Claus-Peter Schnorr and Martin Euchner. Lattice basis reduction: Improved practical
algorithms and solving subset sum problems. Mathematical programming, 66(1-3):181–
199, 1994.

[127] René Schoof. Elliptic curves over finite fields and the computation of square roots mod p.
Mathematics of Computation, 44(170):483–494, 1985.
 125

[128] René Schoof. Counting points on elliptic curves over finite fields. J. Théor. Nombres
Bordeaux, 7(1):219–254, 1995.

[129] Yih-Dar Shieh. Arithmetic aspects of point counting and Frobenius distributions. PhD
thesis, Aix-Marseille, 2015.

[130] G. Shimura. Introduction to the arithmetic theory of automorphic functions, volume 11

of Publications of the Mathematical Society of Japan. Iwanami Shoten and Princeton
University Press, 1971.

[131] Victor Shoup. Lower bounds for discrete logarithms and related problems. In EURO-
CRYPT 1997, volume 1233 of LNCS, pages 256–266. Springer, 1997.

[132] Andrew J. Sommese and Charles W. Wampler II. The numerical solution of systems of
polynomials arising in engineering and science. World Scientific, 2005.

[133] Andrew Sutherland. A generic approach to searching for Jacobians. Mathematics of

Computation, 78(265):485–507, 2009.

[134] Andrew V. Sutherland. Order computations in generic groups. PhD thesis, Massachusetts
Institute of Technology, 2007.

[135] Andrew V. Sutherland. Accelerating the CM method. LMS Journal of Computation and
Mathematics, 15:172–204, 2012.

[136] Agnes Szanto. Multivariate subresultants using Jouanolou matrices. Journal of Pure and
Applied Algebra, 214(8):1347–1369, 2010.

[137] John Tate. Endomorphisms of abelian varieties over finite fields. Inventiones mathemati-
cae, 2(2):134–144, 1966.

[138] Walter Tautz, Jaap Top, and Alain Verberkmoes. Explicit hyperelliptic curves with real
multiplication and permutation polynomials. Canad. J. Math, 43(5):1055–1064, 1991.

[139] Richard Taylor. Automorphy for some `-adic lifts of automorphic mod ` Galois represen-
tations. II. Publications mathématiques, 108(1):183–239, 2008.

[140] Gérald Tenenbaum. Introduction to analytic and probabilistic number theory. Cambridge
university press, 1995.

[141] Akira Terui. Recursive polynomial remainder sequence and its subresultants. Journal of
Algebra, 320(2):633–659, 2008.

[142] Jan Tuitman. Counting points on curves using a map to P1 , II. Finite Fields and Their
Applications, 45:301–322, 2017.

[143] Gilles Villard. On computing the resultant of generic bivariate polynomials. In Proceedings
of ISSAC 2018. ACM, 2018.

[144] Paul van Wamelen. Proving that a genus 2 curve has complex multiplication. Mathematics
of Computation, 68(228):1663–1677, 1999.

[145] William C. Waterhouse and James S. Milne. Abelian varieties over finite fields. In Proc.
Sympos. Pure Math, volume 20, pages 53–64, 1971.
126 Bibliography

[146] Annegret Weng. Constructing hyperelliptic curves of genus 2 suitable for cryptography.
Mathematics of Computation, 72(241):435–458, 2003.

[147] Richard Zippel. Effective polynomial computation. Springer Verlag, 1993.

 Résumé en Français

Courbes algébriques sur les corps finis et applications

Les courbes algébriques font partie du paysage mathématique depuis plus de 2000 ans, depuis
l’Antiquité et les fondements de la Géométrie jusqu’à la preuve du dernier théorème de Fermat
dans les années 1990. De telles courbes sont souvent décrites comme le lieu des solutions d’un sys-
tèmes polynomial et modélisent de nombreuses situations, d’où leur vaste domaine d’application
y compris en dehors des Mathématiques. Dans cette thèse, nous nous concentrons sur les courbes
algébriques planes, c’est-à-dire des courbes données par une équation de la forme f (x, y) = 0,
avec f un polynome bivarié. Un point de la courbe correspond à une solution de son équation,
mais il faut s’accorder sur le sens à donner à la notion de solution. Les mathématiciens grecs de
l’Antiquité en savent quelque chose puisqu’ils ont dû faire face au fait que même des équations à
coefficients entiers peuvent avoir des solutions irrationnelles. Il nous faut donc préciser le corps
dans lequel vivent les coefficients du polynôme f , que nous appellerons le corps de base de la
courbe, et considérer les points de la courbe dans la clôture algébrique de ce corps. Rien ne
s’oppose toutefois à ce que certains d’entre eux appartiennent bien au corps de base, et nous
les qualifierons de rationnels. Bien que le corps des réels semble particulièrement naturel (sans
mauvais jeu de mots) pour étudier les corps et en particulier pour les tracer, les courbes définies
sur les corps finis sont également sources d’intérêt pour leurs multiples applications. Dans cette
thèse, nous considérons presque exclusivement des courbes définies sur un corps fini de carac-
téristique impaire, bien que nous profitons parfois de propriétés de certaines courbes qui sont
des réductions modulo un nombre premier de courbes définies sur Q.
Les courbes sur les corps finis ont historiquement trouvé leurs premières applications en
théorie des nombres, et plus particulièrement pour factoriser des entiers ou tester leur primalité.
En effet, l’algorithme ECM [93] est toujours compétitif par rapport aux algorithmes basé sur
le crible sur les corps de nombres (NFS) pour trouver des “petits” facteurs (de taille inférieure
à 83 bits). L’algorithme ECPP introduit par Goldwasser et Kilian, puis amélioré par Atkin
et Morain [65, 109] est encore aujourd’hui parmi les plus rapides pour générer des certificats
de primalité, il a même récemment été utilisé pour prouver la primalité d’un entier de 34987
bits [76]. Si son efficacité n’est plus à prouver, la complexité de cet algorithme reste cependant
heuristique. Toujours en utilisant des courbes algébriques, de genre 2 cette fois-ci, Adleman
et Huang [4] ont pu construire un algorithme de Las-Vegas pour prouver la primalité en temps
polynomial. Une approche basée sur des courbes encore plus générales est évoquée pour parvenir
à un algorithme de factorisation de polynômes sur des corps finis qui soit à la fois détermin-
iste et de complexité polynomiale. On peut également mentionner l’utilisation de techniques
d’interpolation sur des courbes algébriques par Chudnovsky et Chudnovsky [31] dans les années
1980 pour étudier la complexité du produit de deux polynômes sur des corps finis. Cette ap-
proche est toujours d’actualité et il existe une littérature abondante et récente [118, 9] visant à
construire les meilleures courbes possibles dans cette optique. En particulier, on recherche des

127
128 Résumé en Français

courbes avec autant de points rationnels que possible, par exemple les courbes regroupées dans
la base de donnée libre [63].
Il en va de même pour l’algorithme ECM dont on souhaite améliorer les performances en
choisissant des familles de courbes particulièrement adéquates, soit parce qu’elles ont plus de
chances d’avoir un nombre de points friable [6, 10], ou parce qu’elles permettent une arithmé-
tique plus rapide [108, 16]. Si la factorisation d’entiers intéresse fortement les cryptanalystes,
les courbes algébriques et notamment les courbes elliptiques ont également des applications con-
structives en cryptographie. En effet, le groupe des points rationnels d’une courbe elliptique sur
un corps fini est un parfait exemple de groupe dans lequel calculer des logarithmes discrets est
difficile. Contrairement à RSA et au logarithme discret dans le groupe multiplicatif des corps
finis, il n’existe pas pour le moment d’attaque sous-exponentielle, ce qui permet d’opter pour des
clés bien plus petites. Cela dit, même un algorithme exponentiel peut être efficace si le groupe
utilisé est de petite taille ou si son cardinal est très friable [116]. Compter le nombre de points
rationnels d’une courbe est donc une étape essentielle avant de décider ou non si elle respecte
l’exigence de sécurité que l’on se fixe.
À mesure que l’étude des courbes se développait, d’autres objets mathématiques associés
furent introduits, à l’image des nombreuses fonctions L et zeta qui occupent aujourd’hui une
place centrale dans la théorie des nombres moderne. Ainsi, on trouve plusieurs exemples
d’énoncés de théorie des nombres qui furent établis en prouvant des résultats de nature an-
alytique sur des fonctions complexes, comme par exemple la conjecture de Sato -Tate. Cette
conjecture concerne la distribution du nombre de points rationnels de la réduction modulo p
d’une courbe elliptique définie sur Q lorsque p varie, et fut prouvée autour de 2005 [67, 32, 139].
Des travaux sont en cours pour formuler des conjectures similaires dans des cas plus généraux,
notamment en genre 2 et 3 [49]. Pour ce faire, des algorithmes de comptage de points comme
celui d’Harvey [71] sont au coeur d’expériences numériques impliquant une puissance de calcul
considérable.
Chacune de ces applications a son propre contexte, de la nature des courbes utilisées à la
taille du corps de définition. Dans cette thèse, nous considérons exclusivement des courbes
hyperelliptiques données par un modèle imaginaire y 2 = f (x), avec f un polynôme unitaire sans
carré de degré impair. Le degré deg f = 2g + 1 détermine le genre g de la courbe associée, qui
est un paramètre important dans tout le manuscrit. Deux paramètres supplémentaires p et n
déterminent la caractéristique p et la taille q = pn du corps de base de la courbe. Dans tout le
manuscrit, nous utilisons la notation standard O(), la notation O() e lorsque nous omettons les
termes polylogarithmiques, et la notation Og quand nous omettons également tous les termes
dépendant uniquement de g (et indépendants de q). En utilisant des algorithmes rapides (voir
par exemple [24]), nous partons du principe que chaque opération dans le corps fini Fq a un coût
en O(log
e q).

L’algorithme de Schoof
Nous venons de voir plusieurs raisons pour lesquelles connaître le nombre de points rationnels
d’une courbe elliptique peut être capital. Pour ce faire, une approche est de construire des
courbes ayant un nombre de points spécifié à l’avance, par exemple avec la méthode CM de [6] qui
fut utilisée en cryptographie [86]. Une autre façon de procéder consiste à considérer des courbes
au hasard, compter leurs points rationnels et répéter tant que le résultat n’est pas satisfaisant.
Bien qu’il existe des approches élémentaires pour réaliser cette tâche, comme par exemple tester
toutes les paires (x, y) ∈ Fq pour vérifier si elles satisfont l’équation de la courbe, leur complexité
les rend inadaptées dans la plupart des cas. En 1985, Schoof propose le premier algorithme de
 129

comptage de points dont la complexité est polynomiale en log q [127]. Bien qu’à cette époque
son algorithme ne soit pas considéré suffisamment efficace pour une utilisation pratique, il a
ouvert la voie à de nombreuses améliorations et généralisations aujourd’hui regroupées sous
le terme de méthodes `-adiques. Quelques années plus tard, Elkies et Atkin conçurent des
améliorations [128] qui en firent un algorithme utilisable et remarquablement efficace. Sous le
nom SEA (Schoof-Elkies-Atkin), cette variante de l’algorithme de Schoof permet aujourd’hui
de compter les points d’une courbe elliptique et de générer des courbes cryptographiques de
manière plus que satisfaisante.
L’idée de l’algorithme de Schoof est de calculer le nombre de points rationnels modulo des
nombres premiers ` jusqu’à ce que la valeur exacte puisse être déduite en appliquant le théorème
des restes chinois. En effet, les bornes de Weil impliquent que ce nombre se trouve dans un
√
intervalle de taille d4 qe et donc que le nombre et la taille du plus grand ` à considérer sont
tous les deux en O(log q). Pour obtenir l’information modulo `, Schoof fait agir l’endomorphisme
de Frobenius π : (x, y) 7→ (xq , y q ) sur la `-torsion, c’est-à-dire l’ensemble des points P tels que
`P est le point à l’infini, qui est l’élément neutre pour l’addition sur les points de la courbe. Pour
` premier différent de la caractéristique, la ` torsion est en fait un espace vectoriel isomorphe
à (Z /` Z)2 . L’action de l’endomorphisme de Frobenius est alors représentée par une matrice
2 × 2 dont la trace détermine le nombre de points rationnels modulo `. L’étape la plus coûteuse
dans cet algorithme est le calcul de π dans la `-torsion, qui coûte O(` e 2 log q) opérations dans
le corps de base. En prenant en compte le coût de telles opérations, la taille du plus grand `
et le nombre de premiers ` à considérer, la complexité totale de l’algorithme de Schoof est en
5
O(log
e q). Dans l’algorithme SEA, le gain de complexité est réalisé en remplaçant la `-torsion
par un sous groupe isomorphe à Z /` Z dans lequel chaque opération coûte O(` e log q) opérations
4
dans le corps de base, ce qui donne une complexité en O(log q) pour l’algorithme SEA.
e

Jacobiennes de courbes

Pour des applications telles que la construction de groupes cryptographiques, la notion qui
généralise une courbe elliptique n’est pas directement une courbe de genre plus grand, parce que
ses points rationnels ne forment pas nécessairement un groupe. Les jacobiennes de telles courbes
représentent un objet plus adapté car elles ont une structure de groupe (plus précisément de
variété abélienne). Il en va de même pour la `-torsion qui est celle de la jacobienne et non plus
de la courbe elle-même. Nous verrons qu’en genre plus grand que 2, la détermination de la
`-torsion surpasse l’application du Frobenius et devient l’étape dominante dans la complexité.
Cette étape repose de manière cruciale sur l’arithmétique de la jacobienne.
Bien qu’il existe des algorithmes permettant d’effectuer des additions dans des jacobiennes
de courbes non-hyperelliptiques, par exemple [73, 83], cette thèse se concentre sur le cas hy-
perelliptique parce que cela simplifie grandement l’arithmétique dans les jacobiennes associées,
et notamment la façon dont on parvient à décrire la `-torsion. Les éléments de jacobiennes de
courbes hyperelliptiques de genre g peuvent être représentés par leurs coordonnées de Mumford,
c’est-à-dire par une paire de polynômes de degrés respectifs bornés par g et g − 1. L’addition de
deux éléments sous cette forme est réalisée avec l’algorithme de Cantor [27], avec une complex-
ité en temps et en mémoire qui est quasi-linéaire en g log q. Via une exponentiation binaire, on
déduit de cet algorithme un moyen efficace de multiplier des éléments d’une jacobienne hyper-
elliptique par des scalaires.
130 Résumé en Français

Comptage de points sur des courbes

Dans les années 1990, Pila [114] constate que les résultats théoriques sur lesquels repose l’algorithme
de Schoof sont valides bien au-delà des courbes elliptiques. Il étend ainsi l’algorithme de
Schoof au cas des variétés abéliennes et en particulier des (jacobiennes de) courbes algébriques.
L’algorithme de Pila ne se contente pas de renvoyer le nombre de points rationnels, mais le
polynôme caractéristique de l’endomorphisme de Frobenius ou, de manière équivalente, la fonc-
tion zeta locale de la courbe. Comme l’algorithme de Schoof, l’algorithme de Pila est polynomial
en log q mais il dépend en plus de paramètres comme le genre (la dimension) de la courbe, et d’une
manière exponentielle. Cet algorithme n’a pas été conçu pour être directement implémentable,
mais dans le cas particulier des courbes hyperelliptiques de genre 2, l’emploi de techniques issues
du calcul formel pour décrire les sous-groupes de torsion et l’action du Frobenius sur ces sous-
groupes a permis à Gaudry et Harley [57] de créer et d’implémenter un analogue de l’algorithme
de Schoof. Cet algorithme fut ensuite amélioré par Gaudry et Schost au point d’être suffisam-
ment efficace pour générer une jacobienne de courbe de genre 2 de taille cryptographique [60, 62].
Notons que comme en genre 1, il est toujours possible de créer des jacobiennes avec un nombre
de points fixé à l’avance via la méthode CM [146, 135, 44].
Au début des années 2000, d’autres méthodes également basées sur le calcul de l’action
(d’une approximation p-adique) du Frobenius ont été développées, d’abord par Satoh [125] dans
le cas elliptique. Cette méthode a ensuite été étendue dans un contexte bien plus général et
de nombreux algorithmes, regroupés sous le noms de méthodes p-adiques, ont vu le jour. Ces
différents algorithmes considèrent différents relèvements du Frobenius agissant sur différents es-
paces, comme par exemple celui de Kedlaya [80] basé sur la cohomologie de Monsky-Washnitzer
qui s’applique à des courbes hyperelliptiques de genre quelconque, et son analogue en carac-
téristique 2 par Denef et Vercauteren [41]. D’autres extensions à des courbes de plus en plus
générales ont ensuite été proposées [30, 29, 142] et font toujours l’objet d’une recherche active.
En caractéristique 2, une variante de l’agorithme de Satoh fut proposée indépendamment par
Mestre [102] qui propose une expression du Frobenius en termes de suite artihmético-géométrique
et qui représente aujourd’hui la méthode la plus rapide pour compter les points de courbes ellip-
tiques définies sur F2n . Toujours dans [102], Mestre propose une généralisation de sa méthode
en genre 2, et d’autres travaux l’ont ensuite étendue dans deux directions : soit en choisissant
un corps de (petite) caractéristique impaire [94], soit en considérant des courbes de genre plus
grand [121, 95].
Ces méthodes fournissent des algorithmes utilisables en pratique, et dont la complexité est
polynomiale en g et en n, mais exponentielle en log p, de sorte que les méthodes p-adiques et
`-adiques sont complémentaires lorsque l’un des deux paramètres g et p est petit. En revanche,
il n’existe pas d’algorithme classique de comptage de points dont la complexité est polynomiale
en ces deux paramètres. Notons toutefois que Kedlaya [81] a proposé un tel algorithme en
exploitant des primitives quantiques et que pour une courbe définie sur Q, Harvey [70] parvient
à compter les points de ses réductions modulo tous les nombres premiers p inférieurs à une
borne N en temps quasi-linéaire en N , ce qui veut dire que la complexité moyenne par p est
polynomiale en p. Bien que cela ne s’applique qu’à des réductions d’une même courbe sur Q,
ces algorithmes sont particulièrement adaptés pour formuler des généralisations de la conjecture
de Sato -Tate.
Dans cette thèse, lorsque nous parlons de compter les points d’une courbe ou de sa jacobienne,
nous parlons en réalité de résoudre le problème suivant.
 131

Calcul de la fonction zeta d’une courbe hyperelliptique. Étant donné q

une puissance d’un nombre premier impair, un entier g ≤ 1 et un polynôme univarié
f ∈ Fq [X] de degré 2 g + 1, soit C la courbe hyperelliptique associée au modèle
de Weierstrass Y 2 = f (X). Calculer le numérateur PC ∈ Z[T ] de la fonction zeta
locale de C:
∞
!
X Ti PC (T )
Z(C/Fq , T ) = exp #C(Fqi ) · = .
i=1
i (1 − T )(1 − qT )

Avec C(Fqi ) l’ensemble des points C dont les coordonnée sont dans Fqi .

Sous-groupes de torsion
Une étape clé dans les méthodes `-adiques est la détermination de l’action du Frobenius sur les
sous-groupes de `-torsion. Dans l’algorithme de Schoof, la `-torsion de la courbe est l’ensemble
des points dont l’abscisse annule des polynômes ψ` de degrés (`2 −1)/2 que l’on appelle polynômes
de `-division. Ainsi l’action du Frobenius π : (x, y) 7→ (xq , y q ) peut être calculée en répétant des
étapes d’exponentiation et de réduction par les équations définissant la `-torsion : y 2 = f (x) et
ψ` (x) = 0. Dans un contexte plus général, Pila appelle cette étape le calcul d’une représentation
de bas degré du Frobenius.
Pour les courbes elliptiques, les polynômes de division donnent une représentation simple et
manipulable de la `-torsion. Pour des courbes de genre supérieur, a priori, nous ne pouvons pas
calculer de représentation de bas degré du Frobenius comme dans l’algorithme de Schoof car
nous n’avons de telle description pour la `-torsion. Il faut donc calculer une telle représentation,
par exemple en calculant une base de Gröbner de l’idéal de torsion, avant de pouvoir réduire le
Frobenius.
Dans cette thèse, nous suivons l’approche de Gaudry-Harley-Schost [57, 60, 62] et com-
mençons par écrire l’équation `D = 0 dans la jacobienne. Pour ce faire, nous avons besoin
d’une description de la multiplication par ` en tant qu’application rationnelle. Pour P un point
d’une courbe hyperelliptique, il existe 2g + 2 polynômes décrivant les coordonnées de Mumford
du diviseur `(P − P∞ ). Ces polynômes ont été introduits par Cantor [28] et nommés d’après
lui. En écrivant un élément D de la jacobienne comme somme formelle de points, on peut ainsi
déduire une première description de la `-torsion en tant qu’ensemble des solutions du système
`D = 0.
Une fois ce système calculé, nous le résolvons afin d’avoir une représentation de la `-torsion
nous permettant de réduire le Frobenius. Cette étape est la plus coûteuse dans nos algorithmes,
à la fois en théorie et en pratique. Nous apportons ainsi un soin particulier à la façon dont nous
modélisons la `-torsion par des systèmes polynomiaux et aux techniques de résolution que nous
utilisons car elles ont un impact significatif sur les complexités et les temps de calcul de nos
algorithmes de comptage de points.

Résolution de systèmes polynomiaux

Étant donnés des polynômes multivariés f1 , . . . , fm dans K[x1 , . . . , xn ], nous voulons trouver des
équations définissant l’idéal I = hf1 , . . . , fm i et telles qu’il soit possible d’effectuer des opérations
arithmétiques dans K[x1 , . . . , xn ]/I. Dans cette thèse, nous nous acquittons de cette tâche pour
I = I` , l’idéal de `-torsion d’une jacobienne de courbe hyperelliptique, soit en mettant sous
132 Résumé en Français

forme triangulaire le système f1 = 0, . . . , fm = 0, ou en calculant une résolution géométrique

de ce système, c’est-à-dire une paramétrisation des coordonnées de ses solutions par les racines
d’un polynôme univarié. Dans les deux cas, nous appelons cette opération “résoudre le système”.
La littérature propose de nombreuses façons de résoudre des systèmes polynomiaux et nous en
utiliserons 3 selon leur complexité asymptotique ou leur efficacité pratique. Cela dépend de
nombreux paramètres comme le nombre de variables ou le degré des équations définissant notre
système, mais aussi de propriétés moins évidentes comme la dimension de l’idéal, son degré (le
nombre de solutions dans la clôture algébrique s’il est fini), ainsi que d’éventuelles particularités
structurelles du système. Dans cette thèse, nous résolvons des systèmes qui modélisent des sous-
ensembles de la `-torsion, donc nous savons à l’avance qu’ils sont zéro-dimensionnels et que leur
degré est borné par `2g .
La torsion des courbes de genre 2, par exemple, est représentée par un système d’équations
en deux variables, que l’on peut mettre sous forme triangulaire en calculant des résultants
bivariés, qui représentent actuellement la meilleure option à la fois en termes d’efficacité et de
complexité asymptotique. Dans le chapitre 6, nous modélisons la `-torsion d’une jacobienne
de courbe hyperelliptique de genre 3 par un système polynomial trivarié que nous mettons en
forme triangulaire en calculant des résultants. Bien que cette approche donne une complexité
asymptotique satisfaisante, calculer une base de Gröbner du système via l’algorithme F4 [45]
s’avère bien plus efficace en pratique, aussi est-ce l’approche choisie dans nos expériences. Cela
dit, bien que la complexité des algorithmes F4 et F5 [45, 46] ait été intensément étudiée [11, 12],
aucune borne de complexité présente dans la littérature n’est assez fine pour nous permettre
d’utiliser ces algorithmes en théorie et en pratique.
Dans le chapitre 5, nous modélisons d’une autre manière la `-torsion de courbes hyperel-
liptiques de genre quelconque en faisant intervenir O(g 2 ) variables au lieu de g. Cependant,
il n’y a toujours que g variables dont le degré dépend de `, toutes les autres ayant un degré
en Og (1). Dans ce cas, l’emploi de l’algorithme de résolution géométrique de [26, 64] est dicté
par la nécessité d’invoquer des résultats de complexité qui prennent en compte cette structure
multihomogène si particulière.

Contributions
Dans cette thèse, nous étudions les méthodes `-adiques dérivées des algorithmes de Schoof et Pila.
La complexité de tels algorithmes est au cœur de ce manuscrit, et notamment la dépendance en g
dans l’exposant de log q. La première contribution, publiée en tant que [1], propose un algorithme
de comptage de points sur les courbes hyperelliptiques en grande caractéristique dont on borne
la complexité par une puissance de log q qui croît linéairement en g. Ce résultat s’inscrit dans la
continuité des travaux d’Adleman et Huang [3] qui ont établi que cet exposant était polynomial en
g dans le cas général, et quasi-quadratique dans le cas hyperelliptique. L’état de l’art concernant
cet exposant est résumé dans la table 3. Pour atteindre une telle complexité, notre algorithme
n’est guère différent de celui de Pila, mais notre analyse de complexité fait intervenir une nouvelle
modélisation de la `-torsion par un système polynomial structuré, comme expliqué plus haut.
Cette structure est la clé de voûte de notre résultat, et l’on remarque d’ailleurs qu’en suivant
notre raisonnement sans  l’exploiter, on retrouve un résultat similaire à celui d’Adleman et Huang
O(g 2 log g)
en O (log q) . Si l’idée est naturelle, son exécution nécessite de surmonter quelques
obstacles techniques et notamment de s’assurer que le système polynomial que l’on considère
vérifie bien des hypothèses de généricité sur lesquelles reposent les résultats de complexité pour
le calcul de résolution géométrique. Un autre obstacle est qu’en réalité notre modélisation fait
intervenir un grand nombre de systèmes polynômiaux pour capturer toute la `-torsion, y compris
 133

certains éléments “spéciaux”.

Table 3: Complexité asymptotique pour calculer la fonction zeta locale d’une variété abélienne
de dimension g sur Fq

Auteurs (année) Complexité Contexte

 O(g)

Pila (1990) O (log q)g Variétés abéliennes
 
Huang-Ierardi (1998) O (log q) g O(1) Courbes planes
 
Adleman-Huang (2001) O (log q) g O(1) Variétés abéliennes
 
Adleman-Huang (2001) O (log q) O(g 2 log g) Courbes hyperelliptiques
 
Chapitre 5 (2017) Og (log q)O(g) Courbes hyperelliptiques
8


Chapitre 7 (2018) O
e η (log q) Courbes hyp. avec RM

Un autre aspect que nous étudions concerne l’utilisation pratique d’algorithmes inspirés par
Schoof et Pila lorsque le genre est petit, ce qui va de pair avec la valeur exacte de l’exposant
de log q à genre fixé. Bien que l’algorithme de Pila ne soit pas adapté à une implémentation
directe, ce que Pila appelle une représentation réduite du Frobenius, c’est-à-dire la réduction
du Frobenius modulo l’idéal de `-torsion peut être calculé en pratique à l’aide de techniques
standard issues du calcul formel. En genre 2, c’est précisément ce qui a été réalisé et implé-
menté par Gaudry, Harley et Schost dans [57, 60, 62]. Si la taille des objets à manipuler est
sensiblement plus grande que dans le cas elliptique, cette approche est suffisamment efficace
pour permettre la construction d’une courbe cryptographique de genre 2 définie sur un corps
premier de taille 128 bits. Dans cette thèse, nous proposons une analyse heuristique concernant
la faisabilité d’une telle courbe sur un corps de taille 192 bits, qui nous paraît peu probable
en l’état actuel des choses. Les courbes à multiplication réelle (RM) explicite possèdent une
structure supplémentaire qui permet, en genre 2, de ramener l’exposant de log q de 8 à 5 [59],
atteignant ainsi une complexité semblable à l’algorithme de Schoof.
Une autre contribution de ce manuscrit s’intéresse ainsi aux courbes hyperelliptiques de genre
3 [2]. Cette fois, la taille de la `-torsion rend les expériences pratiquement impossibles dès lors
que ` dépasse 3. Cependant, pour des courbes munies d’une multiplication réelle explicite, les
travaux de [59] s’étendent, moyennant quelques subtilités supplémentaires, avec une complexité
6
asymptotique en O(loge q), donc inférieure à celle du cas général en genre 2. Comme on pouvait
s’y attendre avec une telle complexité, cet algorithme est utilisable en pratique, après quelques
modifications. En particulier, nous calculons la fonction zeta locale d’une courbe hyperelliptique
de genre 3 à multiplication réelle définie sur le corps premier F264 −59 , qui a donc une jacobienne
de 192 bits. Notre algorithme s’adapte aisément en un algorithme de comptage de points sur
des courbes sans multiplication réelle explicite, au prix d’une complexité bien plus grande en
14
O(log
e q), ce qui donne une réponse partielle aux interrogations sur la complexité de l’algorithme
de Schoof-Pila en genre 3. Comme en genre 2, l’étape la plus coûteuse est la résolution du système
polynomial décrivant la `-torsion. Ce système est trivarié mais l’élimination successive à base
de résultants est toujours suffisante pour atteindre une complexité quadratique en le degré de
l’idéal. En pratique, cependant, nous mettons le système sous forme triangulaire en calculant
une base de Gröbner avec les algorithme F4 [45] et FGLM [47]. Cette approche est bien plus
134 Résumé en Français

efficace en pratique, malgré des bornes de complexité théoriques bien plus difficiles à contrôler.

Table 4: Complexité asymptotique pour le calcul de fonctions zeta locales de courbes hyperel-
liptiques de genre ≤ 3

Genre Complexité Auteurs (année)

5
g=1 O(log
e q) Schoof (1985)
4
g=1 O(log
e q) Schoof-Elkies-Atkin (∼ 1990)
8
g=2 O(log
e q) Gaudry-Harley-Schost (∼ 2000)
14
g=3 O(log
e q) Chapitre 6 (2018)
5
g = 2 avec RM O(log
e q) Gaudry-Kohel-Smith (2011)
6
g = 3 with RM O(log
e q) Chapitre 6 (2018)

Puisque la littérature présente de nombreux exemples de (familles de) courbes à multipli-

cation réelle en genre quelconque [87, 23, 43, 101, 138], il est naturel de se demander quelles
améliorations cette structure supplémentaire apporte à la complexité asymptotique lorsque g
croît. Nous avons ainsi étendu les méthodes et les résultats obtenus en genre 2 et 3 pour créer
un algorithm de comptage de points sur des courbes hyperelliptiques à multiplication réelle de
genre arbitrairement grand. L’étape essentielle consiste à calculer non plus des résultants mais
une résolution géométrique des noyaux d’endomorphismes de degré `2 . Pour ce faire, nous adap-
tons la technique du Chapitre 5 qui était alors appliquée au noyau de la multiplication par `,
elle-même de degré `2g en tant qu’endomorphisme. Cette différence affecte notre modélisation
en réduisant les degrés des équations de Og (`3 ) à Og (`3/g ). Ainsi, après vérification que les
hypothèses sont toujours vérifiées et application de l’algorithme de résolution géométrique, nous
atteignons une complexité en Oη ((log q)c ), avec c une constante absolue et Oη dissimulant un
terme dépendant de l’ordre par lequel la courbe a multiplication réelle (et donc également de
g). Cela dit, nous insistons sur le fait que notre algorithme n’est pas polynomial en g et en
log q parce que ce terme reste exponentiel en g. Nous analysons justement les raisons de cette
dépendance exponentielle en g en espérant que de futurs travaux permettront de donner des
bornes plus fines ou de remplacer ces étapes.
 Résumé
Le comptage de points de courbes algébriques est une primitive essentielle en théorie des nombres,
avec des applications en cryptographie, en géométrie arithmétique et pour les codes correcteurs.
Dans cette thèse, nous nous intéressons plus particulièrement au cas de courbes hyperelliptiques
définies sur des corps finis de grande caractéristique p. Dans ce cas de figure, les algorithmes
dérivés de ceux de Schoof et Pila sont actuellement les plus adaptés car leur complexité est
polynomiale en log p. En revanche, la dépendance en le genre g de la courbe est exponentielle
et se fait cruellement sentir même pour g = 3.
Nos contributions consistent principalement à obtenir de nouvelles bornes pour la dépendance
en g de l’exposant de log p. Dans le cas de courbes hyperelliptiques, de précédents travaux
donnaient une borne quasi-quadratique que nous avons pu ramener à linéaire, et même constante
dans le cas très particuliers de familles de courbes dites à multiplication réelle (RM).
En genre 3, nous avons proposé un algorithme inspiré de ceux de Schoof et de Gaudry-
Harley-Schost dont la complexité, en général prohibitive, devient très raisonnable dans le cas de
courbes RM. Nous avons ainsi pu réaliser des expériences pratiques et compter les points d’une
courbe hyperelliptique de genre 3 pour un p de 64 bits.

Mots-clés: Courbes hyperelliptiques, comptage de points, méthodes `-adiques.

Abstract
Counting points on algebraic curves has drawn a lot of attention due to its many applications
from number theory and arithmetic geometry to cryptography and coding theory. In this thesis,
we focus on counting points on hyperelliptic curves over finite fields of large characteristic p. In
this setting, the most suitable algorithms are currently those of Schoof and Pila, because their
complexities are polynomial in log q. However, their dependency in the genus g of the curve is
exponential, and this is already painful even in genus 3.
Our contributions mainly consist of establishing new complexity bounds with a smaller
dependency in g of the exponent of log p. For hyperelliptic curves, previous work showed that
it was quasi-quadratic, and we reduced it to a linear dependency. Restricting to more special
families of hyperelliptic curves with explicit real multiplication (RM), we obtained a constant
bound for this exponent.
In genus 3, we proposed an algorithm based on those of Schoof and Gaudry-Harley-Schost
whose complexity is prohibitive in general, but turns out to be reasonable when the input curves
have explicit RM. In this more favorable case, we were able to count points on a hyperelliptic
curve defined over a 64-bit prime field.

Keywords: Hyperelliptic curves, point counting, `-adic methods.

135
136