Polymath proposal: clearinghouse for crowdsourcing COVID-19 data and data cleaning requests

After some discussion with the applied math research groups here at UCLA (in particular the groups led by Andrea Bertozzi and Deanna Needell), one of the members of these groups, Chris Strohmeier, has produced a proposal for a Polymath project to crowdsource in a single repository (a) a collection of public data sets relating to the COVID-19 pandemic, (b) requests for such data sets, (c) requests for data cleaning of such sets, and (d) submissions of cleaned data sets.  (The proposal can be viewed as a PDF, and is also available on Overleaf).  As mentioned in the proposal, this database would be slightly different in focus than existing data sets such as the COVID-19 data sets hosted on Kaggle, with a focus on producing high quality cleaned data sets.  (Another relevant data set that I am aware of is the SafeGraph aggregated foot traffic data, although this data set, while open, is not quite public as it requires a non-commercial agreement to execute.  Feel free to mention further relevant data sets in the comments.)

This seems like a very interesting and timely proposal to me and I would like to open it up for discussion, for instance by proposing some seed requests for data and data cleaning and to discuss possible platforms that such a repository could be built on.  In the spirit of “building the plane while flying it”, one could begin by creating a basic github repository as a prototype and use the comments in this blog post to handle requests, and then migrate to a more high quality platform once it becomes clear what direction this project might move in.  (For instance one might eventually move beyond data cleaning to more sophisticated types of data analysis.)

UPDATE, Mar 25: a prototype page for such a clearinghouse is now up at this wiki page.

UPDATE, Mar 27: the data cleaning aspect of this project largely duplicates the existing efforts at the United against COVID-19 project, so we are redirecting requests of this type to that project (and specifically to their data discourse page).  The polymath proposal will now refocus on crowdsourcing a list of public data sets relating to the COVID-19 pandemic.

 

Effective approximation of heat flow evolution of the Riemann xi function, and a new upper bound for the de Bruijn-Newman constant

The Polymath15 paper “Effective approximation of heat flow evolution of the Riemann function, and a new upper bound for the de Bruijn-Newman constant“, submitted to Research in the Mathematical Sciences, has just been uploaded to the arXiv. This paper records the mix of theoretical and computational work needed to improve the upper bound on the de Bruijn-Newman constant . This constant can be defined as follows. The function

where is the Riemann function

has a Fourier representation

where is the super-exponentially decaying function

The Riemann hypothesis is equivalent to the claim that all the zeroes of are real. De Bruijn introduced (in different notation) the deformations

of ; one can view this as the solution to the backwards heat equation starting at . From the work of de Bruijn and of Newman, it is known that there exists a real number – the de Bruijn-Newman constant – such that has all zeroes real for and has at least one non-real zero for . In particular, the Riemann hypothesis is equivalent to the assertion . Prior to this paper, the best known bounds for this constant were

with the lower bound due to Rodgers and myself, and the upper bound due to Ki, Kim, and Lee. One of the main results of the paper is to improve the upper bound to

At a purely numerical level this gets “closer” to proving the Riemann hypothesis, but the methods of proof take as input a finite numerical verification of the Riemann hypothesis up to some given height (in our paper we take ) and converts this (and some other numerical verification) to an upper bound on that is of order . As discussed in the final section of the paper, further improvement of the numerical verification of RH would thus lead to modest improvements in the upper bound on , although it does not seem likely that our methods could for instance improve the bound to below without an infeasible amount of computation.

We now discuss the methods of proof. An existing result of de Bruijn shows that if all the zeroes of lie in the strip , then ; we will verify this hypothesis with , thus giving (1). Using the symmetries and the known zero-free regions, it suffices to show that

whenever and .

For large (specifically, ), we use effective numerical approximation to to establish (2), as discussed in a bit more detail below. For smaller values of , the existing numerical verification of the Riemann hypothesis (we use the results of Platt) shows that

for and . The problem though is that this result only controls at time rather than the desired time . To bridge the gap we need to erect a “barrier” that, roughly speaking, verifies that

for , , and ; with a little bit of work this barrier shows that zeroes cannot sneak in from the right of the barrier to the left in order to produce counterexamples to (2) for small .

To enforce this barrier, and to verify (2) for large , we need to approximate for positive . Our starting point is the Riemann-Siegel formula, which roughly speaking is of the shape

where , is an explicit “gamma factor” that decays exponentially in , and is a ratio of gamma functions that is roughly of size . Deforming this by the heat flow gives rise to an approximation roughly of the form

where and are variants of and , , and is an exponent which is roughly . In particular, for positive values of , increases (logarithmically) as increases, and the two sums in the Riemann-Siegel formula become increasingly convergent (even in the face of the slowly increasing coefficients ). For very large values of (in the range for a large absolute constant ), the terms of both sums dominate, and begins to behave in a sinusoidal fashion, with the zeroes “freezing” into an approximate arithmetic progression on the real line much like the zeroes of the sine or cosine functions (we give some asymptotic theorems that formalise this “freezing” effect). This lets one verify (2) for extremely large values of (e.g., ). For slightly less large values of , we first multiply the Riemann-Siegel formula by an “Euler product mollifier” to reduce some of the oscillation in the sum and make the series converge better; we also use a technical variant of the triangle inequality to improve the bounds slightly. These are sufficient to establish (2) for moderately large (say ) with only a modest amount of computational effort (a few seconds after all the optimisations; on my own laptop with very crude code I was able to verify all the computations in a matter of minutes).

The most difficult computational task is the verification of the barrier (3), particularly when is close to zero where the series in (4) converge quite slowly. We first use an Euler product heuristic approximation to to decide where to place the barrier in order to make our numerical approximation to as large in magnitude as possible (so that we can afford to work with a sparser set of mesh points for the numerical verification). In order to efficiently evaluate the sums in (4) for many different values of , we perform a Taylor expansion of the coefficients to factor the sums as combinations of other sums that do not actually depend on and and so can be re-used for multiple choices of after a one-time computation. At the scales we work in, this computation is still quite feasible (a handful of minutes after software and hardware optimisations); if one assumes larger numerical verifications of RH and lowers and to optimise the value of accordingly, one could get down to an upper bound of assuming an enormous numerical verification of RH (up to height about ) and a very large distributed computing project to perform the other numerical verifications.

This post can serve as the (presumably final) thread for the Polymath15 project (continuing this post), to handle any remaining discussion topics for that project.

Polymath15, eleventh thread: Writing up the results, and exploring negative t

This is the eleventh research thread of the Polymath15 project to upper bound the de Bruijn-Newman constant , continuing this post. Discussion of the project of a non-research nature can continue for now in the existing proposal thread. Progress will be summarised at this Polymath wiki page.

There are currently two strands of activity.  One is writing up the paper describing the combination of theoretical and numerical results needed to obtain the new bound .  The latest version of the writeup may be found here, in this directory.  The theoretical side of things have mostly been written up; the main remaining tasks to do right now are

1. giving a more detailed description and illustration of the two major numerical verifications, namely the barrier verification that establishes a zero-free region for for , and the Dirichlet series bound that establishes a zero-free region for ; and
2. giving more detail on the conditional results assuming more numerical verification of RH.

Meanwhile, several of us have been exploring the behaviour of the zeroes of for negative ; this does not directly lead to any new progress on bounding (though there is a good chance that it may simplify the proof of ), but there have been some interesting numerical phenomena uncovered, as summarised in this set of slides.  One phenomenon is that for large negative , many of the complex zeroes begin to organise themselves near the curves

(An example of the agreement between the zeroes and these curves may be found here.)  We now have a (heuristic) theoretical explanation for this; we should have an approximation

in this region (where are defined in equations (11), (15), (17) of the writeup, and the above curves arise from (an approximation of) those locations where two adjacent terms , in this series have equal magnitude (with the other terms being of lower order).

However, we only have a partial explanation at present of the interesting behaviour of the real zeroes at negative t, for instance the surviving zeroes at extremely negative values of appear to lie on the curve where the quantity is close to a half-integer, where

The remaining zeroes exhibit a pattern in coordinates that is approximately 1-periodic in , where

A plot of the zeroes in these coordinates (somewhat truncated due to the numerical range) may be found here.

We do not yet have a total explanation of the phenomena seen in this picture.  It appears that we have an approximation

where is the non-zero multiplier

and

The derivation of this formula may be found in this wiki page.  However our initial attempts to simplify the above approximation further have proven to be somewhat inaccurate numerically (in particular giving an incorrect prediction for the location of zeroes, as seen in this picture).  We are in the process of using numerics to try to resolve the discrepancies (see this page for some code and discussion).

 

Polymath15, tenth thread: numerics update

This is the tenth “research” thread of the Polymath15 project to upper bound the de Bruijn-Newman constant , continuing this post. Discussion of the project of a non-research nature can continue for now in the existing proposal thread. Progress will be summarised at this Polymath wiki page.

Most of the progress since the last thread has been on the numerical side, in which the various techniques to numerically establish zero-free regions to the equation have been streamlined, made faster, and extended to larger heights than were previously possible.  The best bound for now depends on the height to which one is willing to assume the Riemann hypothesis.  Using the conservative verification up to height (slightly larger than) , which has been confirmed by independent work of Platt et al. and Gourdon-Demichel, the best bound remains at .  Using the verification up to height claimed by Gourdon-Demichel, this improves slightly to , and if one assumes the Riemann hypothesis up to height the bound improves to , contingent on a numerical computation that is still underway.   (See the table below the fold for more data of this form.)  This is broadly consistent with the expectation that the bound on should be inversely proportional to the logarithm of the height at which the Riemann hypothesis is verified.

As progress seems to have stabilised, it may be time to transition to the writing phase of the Polymath15 project.  (There are still some interesting research questions to pursue, such as numerically investigating the zeroes of for negative values of , but the writeup does not necessarily have to contain every single direction pursued in the project. If enough additional interesting findings are unearthed then one could always consider writing a second paper, for instance.

Below the fold is the detailed progress report on the numerics by Rudolph Dwars and Kalpesh Muchhal.

Polymath15, ninth thread: going below 0.22?

This is the ninth “research” thread of the Polymath15 project to upper bound the de Bruijn-Newman constant , continuing this post. Discussion of the project of a non-research nature can continue for now in the existing proposal thread. Progress will be summarised at this Polymath wiki page.

We have now tentatively improved the upper bound of the de Bruijn-Newman constant to . Among the technical improvements in our approach, we now are able to use Taylor expansions to efficiently compute the approximation to for many values of in a given region, thus speeding up the computations in the barrier considerably. Also, by using the heuristic that behaves somewhat like the partial Euler product , we were able to find a good location to place the barrier in which is larger than average, hence easier to keep away from zero.

The main remaining bottleneck is that of computing the Euler mollifier bounds that keep bounded away from zero for larger values of beyond the barrier. In going below we are beginning to need quite complicated mollifiers with somewhat poor tail behavior; we may be reaching the point where none of our bounds will succeed in keeping bounded away from zero, so we may be close to the natural limits of our methods.

Participants are also welcome to add any further summaries of the situation in the comments below.

Polymath16 now launched: simplifying the lower bound argument for the Hadwiger-Nelson problem

Just a quick announcement that Dustin Mixon and Aubrey de Grey have just launched the Polymath16 project over at Dustin’s blog.  The main goal of this project is to simplify the recent proof by Aubrey de Grey that the chromatic number of the unit distance graph of the plane is at least 5, thus making progress on the Hadwiger-Nelson problem.  The current proof is computer assisted (in particular it requires one to control the possible 4-colorings of a certain graph with over a thousand vertices), but one of the aims of the project is to reduce the amount of computer assistance needed to verify the proof; already a number of such reductions have been found.  See also this blog post where the polymath project was proposed, as well as the wiki page for the project.  Non-technical discussion of the project will continue at the proposal blog post.

Polymath15, seventh thread: going below 0.48

This is the seventh “research” thread of the Polymath15 project to upper bound the de Bruijn-Newman constant , continuing this post. Discussion of the project of a non-research nature can continue for now in the existing proposal thread. Progress will be summarised at this Polymath wiki page.

The most recent news is that we appear to have completed the verification that is free of zeroes when and , which implies that . For very large (for instance when the quantity is at least ) this can be done analytically; for medium values of (say when is between and ) this can be done by numerically evaluating a fast approximation to and using the argument principle in a rectangle; and most recently it appears that we can also handle small values of , in part due to some new, and significantly faster, numerical ways to evaluate in this range.

One obvious thing to do now is to experiment with lowering the parameters and and see what happens. However there are two other potential ways to bound which may also be numerically feasible. One approach is based on trying to exclude zeroes of in a region of the form , and for some moderately large (this acts as a “barrier” to prevent zeroes from flowing into the region at time , assuming that they were not already there at time ). This require significantly less numerical verification in the aspect, but more numerical verification in the aspect, so it is not yet clear whether this is a net win.

Another, rather different approach, is to study the evolution of statistics such as over time. One has fairly good control on such quantities at time zero, and their time derivative looks somewhat manageable, so one may be able to still have good control on this quantity at later times . However for this approach to work, one needs an effective version of the Riemann-von Mangoldt formula for , which at present is only available asymptotically (or at time ). This approach may be able to avoid almost all numerical computation, except for numerical verification of the Riemann hypothesis, for which we can appeal to existing literature.

Participants are also welcome to add any further summaries of the situation in the comments below.

Polymath15, sixth thread: the test problem and beyond

This is the sixth “research” thread of the Polymath15 project to upper bound the de Bruijn-Newman constant , continuing this post. Discussion of the project of a non-research nature can continue for now in the existing proposal thread. Progress will be summarised at this Polymath wiki page.

The last two threads have been focused primarily on the test problem of showing that whenever . We have been able to prove this for most regimes of , or equivalently for most regimes of the natural number parameter . In many of these regimes, a certain explicit approximation to was used, together with a non-zero normalising factor ; see the wiki for definitions. The explicit upper bound

has been proven for certain explicit expressions (see here) depending on . In particular, if satisfies the inequality

then is non-vanishing thanks to the triangle inequality. (In principle we have an even more accurate approximation available, but it is looking like we will not need it for this test problem at least.)

We have explicit upper bounds on , , ; see this wiki page for details. They are tabulated in the range here. For , the upper bound for is monotone decreasing, and is in particular bounded by , while and are known to be bounded by and respectively (see here).

Meanwhile, the quantity can be lower bounded by

for certain explicit coefficients and an explicit complex number . Using the triangle inequality to lower bound this by

we can obtain a lower bound of for , which settles the test problem in this regime. One can get more efficient lower bounds by multiplying both Dirichlet series by a suitable Euler product mollifier; we have found for to be good choices to get a variety of further lower bounds depending only on , see this table and this wiki page. Comparing this against our tabulated upper bounds for the error terms we can handle the range .

In the range , we have been able to obtain a suitable lower bound (where exceeds the upper bound for ) by numerically evaluating at a mesh of points for each choice of , with the mesh spacing being adaptive and determined by and an upper bound for the derivative of ; the data is available here.

This leaves the final range (roughly corresponding to ). Here we can numerically evaluate to high accuracy at a fine mesh (see the data here), but to fill in the mesh we need good upper bounds on . It seems that we can get reasonable estimates using some contour shifting from the original definition of (see here). We are close to finishing off this remaining region and thus solving the toy problem.

Beyond this, we need to figure out how to show that for as well. General theory lets one do this for , leaving the region . The analytic theory that handles and should also handle this region; for presumably the argument principle will become relevant.

The full argument also needs to be streamlined and organised; right now it sprawls over many wiki pages and github code files. (A very preliminary writeup attempt has begun here). We should also see if there is much hope of extending the methods to push much beyond the bound of that we would get from the above calculations. This would also be a good time to start discussing whether to move to the writing phase of the project, or whether there are still fruitful research directions for the project to explore.

Participants are also welcome to add any further summaries of the situation in the comments below.

Polymath15, fifth thread: finishing off the test problem?

This is the fifth “research” thread of the Polymath15 project to upper bound the de Bruijn-Newman constant , continuing this post. Discussion of the project of a non-research nature can continue for now in the existing proposal thread. Progress will be summarised at this Polymath wiki page.

We have almost finished off the test problem of showing that whenever . We have two useful approximations for , which we have denoted and , and a normalising quantity that is asymptotically equal to the above expressions; see the wiki page for definitions. In practice, the approximation seems to be accurate within about one or two significant figures, whilst the approximation is accurate to about three or four. We have an effective upper bound

where the expressions are quite small in practice ( is typically about two orders of magnitude smaller than the main term once is moderately large, and the error terms are even smaller). See this page for details. In principle we could also obtain an effective upper bound for (the term would be replaced by something smaller).

The ratio takes the form of a difference of two Dirichlet series, where is a phase whose value is explicit but perhaps not terribly important, and the coefficients are explicit and relatively simple ( is , and is approximately ). To bound this away from zero, we have found it advantageous to mollify this difference by multiplying by an Euler product to cancel much of the initial oscillation; also one can take advantage of the fact that the are real and the are (approximately) real. See this page for details. The upshot is that we seem to be getting good lower bounds for the size of this difference of Dirichlet series starting from about or so. The error terms are already quite small by this stage, so we should soon be able to rigorously keep from vanishing at this point. We also have a scheme for lower bounding the difference of Dirichlet series below this range, though it is not clear at present how far we can continue this before the error terms become unmanageable. For very small we may have to explore some faster ways to compute the expression , which is still difficult to compute directly with high accuracy. One will also need to bound the somewhat unwieldy expressions by something more manageable. For instance, right now these quantities depend on the continuous variable ; it would be preferable to have a quantity that depends only on the parameter , as this could be computed numerically for all in the remaining range of interest quite quickly.

As before, any other mathematical discussion related to the project is also welcome here, for instance any summaries of previous discussion that was not covered in this post.

Polymath15, fourth thread: closing in on the test problem

This is the fourth “research” thread of the Polymath15 project to upper bound the de Bruijn-Newman constant , continuing https://terrytao.wordpress.com/2018/01/24/polymath-proposal-upper-bounding-the-de-bruijn-newman-constant/. Progress will be summarised at this Polymath wiki page.

We are getting closer to finishing off the following test problem: can one show that whenever , ? This would morally show that . A wiki page for this problem has now been created here. We have obtained a number of approximations to (see wiki page), though numeric evidence indicates that the approximations are all very close to each other. (Many of these approximations come with a correction term , but thus far it seems that we may be able to avoid having to use this refinement to the approximations.) The effective approximation also comes with an effective error bound

for some explicit (but somewhat messy) error terms : see this wiki page for details. The original approximations can be considered deprecated at this point in favour of the (slightly more complicated) approximation ; the approximation is a simplified version of which is not quite as accurate but might be useful for testing purposes.

It is convenient to normalise everything by an explicit non-zero factor . Asymptotically, converges to 1; numerically, it appears that its magnitude (and also its real part) stays roughly between 0.4 and 3 in the range , and we seem to be able to keep it (or at least the toy counterpart ) away from zero starting from about (here it seems that there is a useful trick of multiplying by Euler-type factors like to cancel off some of the oscillation). Also, the bounds on the error seem to be of size about 0.1 or better in these ranges also. So we seem to be on track to be able to rigorously eliminate zeroes starting from about or so. We have not discussed too much what to do with the small values of ; at some point our effective error bounds will become unusable, and we may have to find some more faster ways to compute .

In addition to this main direction of inquiry, there have been additional discussions on the dynamics of zeroes, and some numerical investigations of the behaviour Lehmer pairs under heat flow. Contributors are welcome to summarise any findings from these discussions from previous threads (or on any other related topic, e.g. improvements in the code) in the comments below.