There are multiple purposes to this blog post.

The first purpose is to announce the uploading of the paper “New equidistribution estimates of Zhang type, and bounded gaps between primes” by D.H.J. Polymath, which is the main output of the Polymath8a project on bounded gaps between primes, to the arXiv, and to describe the main results of this paper below the fold.

The second purpose is to roll over the previous thread on all remaining Polymath8a-related matters (e.g. updates on the submission status of the paper) to a fresh thread. (Discussion of the ongoing Polymath8b project is however being kept on a separate thread, to try to reduce confusion.)

The final purpose of this post is to coordinate the writing of a retrospective article on the Polymath8 experience, which has been solicited for the Newsletter of the European Mathematical Society. I suppose that this could encompass both the Polymath8a and Polymath8b projects, even though the second one is still ongoing (but I think we will soon be entering the endgame there). I think there would be two main purposes of such a retrospective article. The first one would be to tell a story about the process of conducting mathematical research, rather than just describe the outcome of such research; this is an important aspect of the subject which is given almost no attention in most mathematical writing, and it would be good to be able to capture some sense of this process while memories are still relatively fresh. The other would be to draw some tentative conclusions with regards to what the strengths and weaknesses of a Polymath project are, and how appropriate such a format would be for other mathematical problems than bounded gaps between primes. In my opinion, the bounded gaps problem had some fairly unique features that made it particularly amenable to a Polymath project, such as (a) a high level of interest amongst the mathematical community in the problem; (b) a very focused objective (“improve {H}!”), which naturally provided an obvious metric to measure progress; (c) the modular nature of the project, which allowed for people to focus on one aspect of the problem only, and still make contributions to the final goal; and (d) a very reasonable level of ambition (for instance, we did not attempt to prove the twin prime conjecture, which in my opinion would make a terrible Polymath project at our current level of mathematical technology). This is not an exhaustive list of helpful features of the problem; I would welcome other diagnoses of the project by other participants.

With these two objectives in mind, I propose a format for the retrospective article consisting of a brief introduction to the polymath concept in general and the polymath8 project in particular, followed by a collection of essentially independent contributions by different participants on their own experiences and thoughts. Finally we could have a conclusion section in which we make some general remarks on the polymath project (such as the remarks above). I’ve started a dropbox subfolder for this article (currently in a very skeletal outline form only), and will begin writing a section on my own experiences; other participants are of course encouraged to add their own sections (it is probably best to create separate files for these, and then input them into the main file retrospective.tex, to reduce edit conflicts. If there are participants who wish to contribute but do not currently have access to the Dropbox folder, please email me and I will try to have you added (or else you can supply your thoughts by email, or in the comments to this post; we may have a section for shorter miscellaneous comments from more casual participants, for people who don’t wish to write a lengthy essay on the subject).

As for deadlines, the EMS Newsletter would like a submitted article by mid-April in order to make the June issue, but in the worst case, it will just be held over until the issue after that.

— 1. Description of Polymath8a results —

Let {H_1} denote the quantity

\displaystyle H_1 := \liminf_{n \rightarrow \infty} p_{n+1}-p_n,

where {p_n} denotes the {n^{th}} prime. Thus for instance the notorious twin prime conjecture is equivalent to the claim that {H_1=2}. However, even establishing the finite nature of {H_1} unconditionally was an open problem until the celebrated work of Zhang last year, who established the bound

\displaystyle H_1 \leq 70,000,000.

Zhang’s argument, which built upon earlier work of Goldston, Pintz, and Yildirim, can be summarised as follows. For any natural number {k}, define an admissible {k}-tuple to be a tuple {(h_1,\dots,h_k)} of increasing integers, which avoids at least one residue class modulo {p} for each prime {p}. For instance, {(0,2,6)} is an admissible {3}-tuple, but {(0,2,4)} is not. The Hardy-Littlewood prime tuples conjecture asserts that if {(h_1,\dots,h_k)} is an admissible {k}-tuple, then there exists infinitely many {n} such that {n+h_1,\dots,n+h_k} are simultaneously prime. This conjecture is currently out of reach for any {k \geq 2}; for instance, the case when {k=2} and the tuple is {(0,2)} is the twin prime conjecture. However, Zhang was able to prove a weaker claim, which we call {DHL[k,2]}, for sufficiently large {k}. Specifically, (following the notation of Pintz) let {DHL[k,2]} denote the assertion that given any admissible {k}-tuple {(h_1,\dots,h_k)}, one has infinitely many {n} such that at least two of the {n+h_1,\dots,n+h_k} are prime. It is easy to see that if {DHL[k,2]} holds and {(h_1,\dots,h_k)} is an admissible {k}-tuple, then {H_1 \leq h_k-h_1}. So to bound {H_1}, it suffices to show that {DHL[k,2]} holds for some {k}, and then find as narrow an admissible {k}-tuple as possible.

Zhang was able to obtain {DHL[k,2]} for {k=3,500,000}, and then took the first {k} primes larger than {k} to be the admissible {k}-tuple, observing that this tuple had diameter at most {70,000,000}. (Actually, it has diameter {59,874,594}, as observed by Trudgian.) The earliest phase of the Polymath8a project consisted of using increasingly sophisticated methods to search for narrow admissible tuples of a given cardinality; in the case of this particular {k}, we were able to find an admissible tuple whose diameter was {55,223,504}. On the other hand, an application of the large sieve inequalities shows that admissible {k}-tuples asymptotically must have size at least {(\frac{1}{2}+o(1)) k\log k} (and we conjecture that the narrowest {k}-tuple in fact has size {(1+o(1)) k \log k}), so there is a definite limit to how much one can improve the bound on {H_1} purely from finding ever narrower admissible tuples. (As part of the Polymath8a project, a database of narrow tuples was set up here (and is still accepting submissions), building upon previous data of Engelsma.)

To make further progress, one has to analyse how the result {DHL[k,2]} is proven. Here, Zhang follows the arguments of Goldston, Pintz, and Yildirim, which are based on constructing a sieve function {\nu: {\bf Z} \rightarrow {\bf R}^+}, supported on (say) the interval {[x,2x]} for a large {x}, such that the sum

\displaystyle \sum_n \nu(n) \ \ \ \ \ (1)

has good upper bounds, and the sums

\displaystyle \sum_n \nu(n) 1_{n+h_i \hbox{ prime}} \ \ \ \ \ (2)

has good lower bounds for {i=1,\dots,k}. Provided that the ratio between the lower and upper bounds is big enough, one can then easily deduce {DHL[k,2]} (essentially from the pigeonhole principle).

One then needs to find a good choice of {\nu}, which on the one hand is simple enough that the sums (1), (2) can be bounded rigorously, but on the other hand are sophisticated enough that one gets a good ratio between (2) and (1). Goldston, Pintz, and Yildirim eventually settled on a choice essentially of the form

\displaystyle \nu(n) = 1_{[x,2x]} (\sum_{d|n: d \leq R} \mu(d) \log^{k+l}(\frac{\log d}{\log R}))^2

for some auxiliary parameter {l \geq 0} and some {R = x^{\theta/2}}; this is a variant of the Selberg sieve. With this choice, they were already able to establish upper bounds of {H_1} as strong as {16} on the Elliott-Halberstam conjecture, which asserts that

\displaystyle \sum_{q \leq x^\theta} \sup_{a \in ({\bf Z}/q{\bf Z})^\times} |\sum_{n \in [x,2x]; n=a\ (q)} \Lambda(n) - \frac{1}{\phi(q)} \sum_{n \in [x,2x]} \Lambda(n)| \ll x\log^{-A} x \ \ \ \ \ (3)

for all {0 < \theta < 1} and {A >0} and to obtain the weaker result {\liminf_{n \rightarrow \infty} \frac{p_{n+1}-p_n}{\log p_n} = o(1)} without this conjecture. Furthermore, any nontrivial progress on the Elliott-Halberstam conjecture (beyond what is provided by the Bombieri-Vinogradov theorem, which covers the case {0 < \theta <1/2}) would give some finite bound on {H_1}.

Even after all the recent progress on bounded gaps, we still do not have any direct progress on the Elliott-Halberstam conjecture (3) for any {\theta \geq 1/2}. However, Zhang (and independently, Motohashi and Pintz) observed that one does not need the full strength of (3) in order to obtain the conclusions of Goldston-Pintz-Yildirim. Firstly, one does not need all residue classes {a \in ({\bf Z}/q{\bf Z})} here, but only those classes that are the roots of a certain polynomial. Secondly, one does not need all moduli here, but can restrict attention to smooth (or friable moduli – moduli with no large prime factors – as the error incurred by ignoring all other moduli turns out to be exponentially small in {k}. With these caveats, Zhang was able to obtain a restricted form of (3) with {\theta} as large as {\frac{1}{2} + \frac{2}{1168}}, which he then used to obtain {k} as small as {3,500,000}.

Actually, Zhang’s treatment of the truncation error is not optimal, and by being more careful here (and by relaxing the requirement of smooth moduli to the less stringent requirement of “densely divisible” moduli) we were able to reduce {k} down to {341,640}. Furthermore, by replacing the monomial {\log^{k+l}(\frac{\log d}{\log R})} with the more flexible cutoff {f( \frac{\log d}{\log R})} and then optimising in {f} (a computation first made in unpublished work of Conrey, and then in the paper of Farkas, Pintz, and Revesz, with the optimal {f} turning out to come from a Bessel function), one could reduce {k} to be as small as {34,429} (leading to a bound of {H_1} that ended up to be {386,344}).

To go beyond this, we had to unpack Zhang’s proof of (a weakened version of) the Elliott-Halberstam type bound (3). His approach follows a well known sequence of papers by Bombieri, Fouvry, Friedlander, and Iwaniec on various restricted breakthroughs beyond the Bombieri-Vinogradov barrier, although with the key difference that Zhang did not use automorphic form techniques, which (at our current level of understanding) are almost entirely restricted to the regime where the residue class {a} is fixed in {q} (as opposed to varying amongst the roots of a polynomial modulo {q}, which is what is needed for the current application). However, the remaining steps are familiar: first one uses the Heath-Brown identity to decompose (a variant of) the expression in (3) into some simpler bilinear and trilinear sums, which Zhang called “Type I”, “Type II”, and “Type III” (though one should caution that these are slightly different from the “Type I” and “Type II” sums arising from Vaughan-type identities). The Type I and Type II sums turn out to be treatable using a careful combination of the Cauchy-Schwarz inequality (as embodied in tools such as the dispersion method of Linnik), the Polya-Vinogradov completion of sums method, and estimates on one-dimensional exponential sums (which are variants of Kloosterman sums) which can ultimately be handled by the Riemann hypothesis for curves over finite fields, first established by Weil (and which can in this particular context also be proven by the elementary method of Stepanov). The Type III sums can be treated by a variant of these methods, except that one-dimensional exponential sum estimates are insufficient; Zhang instead needed to turn to the three-dimensional exponential sum estimates of Birch and Bombieri to get an adequate amount of cancellation, and these estimates ultimately arose from the deep work of Deligne on the Riemann hypothesis for higher dimensional varieties (see this previous blog post for a discussion of these hypotheses).

In our work, we were able to improve the Cauchy-Schwarz components of these arguments in a number of ways, with the most significant gain coming from applying the “{q}-van der Corput {A}-process” of Graham and Ringrose to the Type I sums; we also have a slightly different way to handle the Type III sums (based on a recent preprint of Fouvry, Kowalski, and Michel), based on correlations of hyper-Kloosterman sums (again coming from Deligne’s work), which gives significantly better results for these sums (so much so, in fact, that the Type III sums are no longer the dominant obstruction to further improvement of the numerology). Putting all these computations together, we can stretch Zhang’s improvement to Bombieri-Vinogradov by about an order of magnitude, with {\theta} now allowed to be as large as {\frac{1}{2} + \frac{7}{300} = 0.5233\dots} rather than {\frac{1}{2} + \frac{2}{1158} = 0.5017\dots}. This leads to a value of {k} as low as {632}, which in turn leads to the bound {H_1 \leq 4680}. These latter bounds have since been improved by Maynard and by Polymath8b, mostly by significant improvements to the sieve-theoretic part of the argument (and no longer using any distributional result on the primes beyond the Bombieri-Vinogradov theorem), but the distribution result of Polymath8a is still the best distribution result known on the primes, and may well have other applications beyond the bounded gaps problem.

Interestingly, the {q}-van der Corput {A}-process is strong enough, in fact, that we can still get non-trivial bounds of (weakened versions of) the form (3) even if we don’t attempt to estimate the Type III sums, so in particular we can obtain a Zhang-type distribution theorem even without using Deligne’s theorems, with {\theta} now reaching as large as {\frac{1}{2}+\frac{2}{168} = 0.5119\dots}.