In July I will be spending a week at Park City, being one of the mini-course lecturers in the Graduate Summer School component of the Park City Summer Session on random matrices.  I have chosen to give some lectures on least singular values of random matrices, the circular law, and the Lindeberg exchange method in random matrix theory; this is a slightly different set of topics than I had initially advertised (which was instead about the Lindeberg exchange method and the local relaxation flow method), but after consulting with the other mini-course lecturers I felt that this would be a more complementary set of topics.  I have uploaded an draft of my lecture notes (some portion of which is derived from my monograph on the subject); as always, comments and corrections are welcome.

<I>[Update, June 23: notes revised and reformatted to PCMI format. -T.]</I>

Suppose ${F: X \rightarrow Y}$ is a continuous (but nonlinear) map from one normed vector space ${X}$ to another ${Y}$. The continuity means, roughly speaking, that if ${x_0, x \in X}$ are such that ${\|x-x_0\|_X}$ is small, then ${\|F(x)-F(x_0)\|_Y}$ is also small (though the precise notion of “smallness” may depend on ${x}$ or ${x_0}$, particularly if ${F}$ is not known to be uniformly continuous). If ${F}$ is known to be differentiable (in, say, the Fréchet sense), then we in fact have a linear bound of the form

$\displaystyle \|F(x)-F(x_0)\|_Y \leq C(x_0) \|x-x_0\|_X$

for some ${C(x_0)}$ depending on ${x_0}$, if ${\|x-x_0\|_X}$ is small enough; one can of course make ${C(x_0)}$ independent of ${x_0}$ (and drop the smallness condition) if ${F}$ is known instead to be Lipschitz continuous.

In many applications in analysis, one would like more explicit and quantitative bounds that estimate quantities like ${\|F(x)-F(x_0)\|_Y}$ in terms of quantities like ${\|x-x_0\|_X}$. There are a number of ways to do this. First of all, there is of course the trivial estimate arising from the triangle inequality:

$\displaystyle \|F(x)-F(x_0)\|_Y \leq \|F(x)\|_Y + \|F(x_0)\|_Y. \ \ \ \ \ (1)$

This estimate is usually not very good when ${x}$ and ${x_0}$ are close together. However, when ${x}$ and ${x_0}$ are far apart, this estimate can be more or less sharp. For instance, if the magnitude of ${F}$ varies so much from ${x_0}$ to ${x}$ that ${\|F(x)\|_Y}$ is more than (say) twice that of ${\|F(x_0)\|_Y}$, or vice versa, then (1) is sharp up to a multiplicative constant. Also, if ${F}$ is oscillatory in nature, and the distance between ${x}$ and ${x_0}$ exceeds the “wavelength” of the oscillation of ${F}$ at ${x_0}$ (or at ${x}$), then one also typically expects (1) to be close to sharp. Conversely, if ${F}$ does not vary much in magnitude from ${x_0}$ to ${x}$, and the distance between ${x}$ and ${x_0}$ is less than the wavelength of any oscillation present in ${F}$, one expects to be able to improve upon (1).

When ${F}$ is relatively simple in form, one can sometimes proceed simply by substituting ${x = x_0 + h}$. For instance, if ${F: R \rightarrow R}$ is the squaring function ${F(x) = x^2}$ in a commutative ring ${R}$, one has

$\displaystyle F(x_0+h) = (x_0+h)^2 = x_0^2 + 2x_0 h+ h^2$

and thus

$\displaystyle F(x_0+h) - F(x_0) = 2x_0 h + h^2$

or in terms of the original variables ${x, x_0}$ one has

$\displaystyle F(x) - F(x_0) = 2 x_0 (x-x_0) + (x-x_0)^2.$

If the ring ${R}$ is not commutative, one has to modify this to

$\displaystyle F(x) - F(x_0) = x_0 (x-x_0) + (x-x_0) x_0 + (x-x_0)^2.$

Thus, for instance, if ${A, B}$ are ${n \times n}$ matrices and ${\| \|_{op}}$ denotes the operator norm, one sees from the triangle inequality and the sub-multiplicativity ${\| AB\|_{op} \leq \| A \|_{op} \|B\|_{op}}$ of operator norm that

$\displaystyle \| A^2 - B^2 \|_{op} \leq \| A - B \|_{op} ( 2 \|B\|_{op} + \|A - B \|_{op} ). \ \ \ \ \ (2)$

If ${F(x)}$ involves ${x}$ (or various components of ${x}$) in several places, one can sometimes get a good estimate by “swapping” ${x}$ with ${x_0}$ at each of the places in turn, using a telescoping series. For instance, if we again use the squaring function ${F(x) = x^2 = x x}$ in a non-commutative ring, we have

$\displaystyle F(x) - F(x_0) = x x - x_0 x_0$

$\displaystyle = (x x - x_0 x) + (x_0 x - x_0 x_0)$

$\displaystyle = (x-x_0) x + x_0 (x-x_0)$

which for instance leads to a slight improvement of (2):

$\displaystyle \| A^2 - B^2 \|_{op} \leq \| A - B \|_{op} ( \| A\|_{op} + \|B\|_{op} ).$

More generally, for any natural number ${n}$, one has the identity

$\displaystyle x^n - x_0^n = (x-x_0) (x^{n-1} + x^{n-2} x_0 + \dots + x x_0^{n-2} + x_0^{n-1}) \ \ \ \ \ (3)$

in a commutative ring, while in a non-commutative ring one must modify this to

$\displaystyle x^n - x_0^n = \sum_{i=0}^{n-1} x_0^i (x-x_0) x^{n-1-i},$

and for matrices one has

$\displaystyle \| A^n - B^n \|_{op} \leq \| A-B\|_{op} ( \|A\|_{op}^{n-1} + \| A\|_{op}^{n-2} \| B\|_{op} + \dots + \|B\|_{op}^{n-1} ).$

Exercise 1 If ${U}$ and ${V}$ are unitary ${n \times n}$ matrices, show that the commutator ${[U,V] := U V U^{-1} V^{-1}}$ obeys the inequality

$\displaystyle \| [U,V] - I \|_{op} \leq 2 \| U - I \|_{op} \| V - I \|_{op}.$

(Hint: first control ${\| UV - VU \|_{op}}$.)

Now suppose (for simplicity) that ${F: {\bf R}^d \rightarrow {\bf R}^{d'}}$ is a map between Euclidean spaces. If ${F}$ is continuously differentiable, then one can use the fundamental theorem of calculus to write

$\displaystyle F(x) - F(x_0) = \int_0^1 \frac{d}{dt} F( \gamma(t) )\ dt$

where ${\gamma: [0,1] \rightarrow Y}$ is any continuously differentiable path from ${x_0}$ to ${x}$. For instance, if one uses the straight line path ${\gamma(t) := (1-t) x_0 + tx}$, one has

$\displaystyle F(x) - F(x_0) = \int_0^1 ((x-x_0) \cdot \nabla F)( (1-t) x_0 + t x )\ dt.$

In the one-dimensional case ${d=1}$, this simplifies to

$\displaystyle F(x) - F(x_0) = (x-x_0) \int_0^1 F'( (1-t) x_0 + t x )\ dt. \ \ \ \ \ (4)$

Among other things, this immediately implies the factor theorem for ${C^k}$ functions: if ${F}$ is a ${C^k({\bf R})}$ function for some ${k \geq 1}$ that vanishes at some point ${x_0}$, then ${F(x)}$ factors as the product of ${x-x_0}$ and some ${C^{k-1}}$ function ${G}$. Another basic consequence is that if ${\nabla F}$ is uniformly bounded in magnitude by some constant ${C}$, then ${F}$ is Lipschitz continuous with the same constant ${C}$.

Applying (4) to the power function ${x \mapsto x^n}$, we obtain the identity

$\displaystyle x^n - x_0^n = n (x-x_0) \int_0^1 ((1-t) x_0 + t x)^{n-1}\ dt \ \ \ \ \ (5)$

which can be compared with (3). Indeed, for ${x_0}$ and ${x}$ close to ${1}$, one can use logarithms and Taylor expansion to arrive at the approximation ${((1-t) x_0 + t x)^{n-1} \approx x_0^{(1-t) (n-1)} x^{t(n-1)}}$, so (3) behaves a little like a Riemann sum approximation to (5).

Exercise 2 For each ${i=1,\dots,n}$, let ${X^{(1)}_i}$ and ${X^{(0)}_i}$ be random variables taking values in a measurable space ${R_i}$, and let ${F: R_1 \times \dots \times R_n \rightarrow {\bf R}^m}$ be a bounded measurable function.

• (i) (Lindeberg exchange identity) Show that

$\displaystyle \mathop{\bf E} F(X^{(1)}_1,\dots,X^{(1)}_n) - \mathop{\bf E} F(X^{(0)}_1,\dots,X^{(0)}_n)$

$\displaystyle = \sum_{i=1}^n \mathop{\bf E} F( X^{(1)}_1,\dots, X^{(1)}_{i-1}, X^{(1)}_i, X^{(0)}_{i+1}, \dots, X^{(0)}_n)$

$\displaystyle - \mathop{\bf E} F( X^{(1)}_1,\dots, X^{(1)}_{i-1}, X^{(0)}_i, X^{(0)}_{i+1}, \dots, X^{(0)}_n).$

• (ii) (Knowles-Yin exchange identity) Show that

$\displaystyle \mathop{\bf E} F(X^{(1)}_1,\dots,X^{(1)}_n) - \mathop{\bf E} F(X^{(0)}_1,\dots,X^{(0)}_n)$

$\displaystyle = \int_0^1 \sum_{i=1}^n \mathop{\bf E} F( X^{(t)}_1,\dots, X^{(t)}_{i-1}, X^{(1)}_i, X^{(t)}_{i+1}, \dots, X^{(t)}_n)$

$\displaystyle - \mathop{\bf E} F( X^{(t)}_1,\dots, X^{(t)}_{i-1}, X^{(0)}_i, X^{(t)}_{i+1}, \dots, X^{(t)}_n)\ dt,$

where ${X^{(t)}_i = 1_{I_i \leq t} X^{(0)}_i + 1_{I_i > t} X^{(1)}_i}$ is a mixture of ${X^{(0)}_i}$ and ${X^{(1)}_i}$, with ${I_1,\dots,I_n}$ uniformly drawn from ${[0,1]}$ independently of each other and of the ${X^{(0)}_1,\dots,X^{(0)}_n, X^{(1)}_0,\dots,X^{(1)}_n}$.

• (iii) Discuss the relationship between the identities in parts (i), (ii) with the identities (3), (5).

(The identity in (i) is the starting point for the Lindeberg exchange method in probability theory, discussed for instance in this previous post. The identity in (ii) can also be used in the Lindeberg exchange method; the terms in the right-hand side are slightly more symmetric in the indices ${1,\dots,n}$, which can be a technical advantage in some applications; see this paper of Knowles and Yin for an instance of this.)

Exercise 3 If ${F: {\bf R}^d \rightarrow {\bf R}^{d'}}$ is continuously ${k}$ times differentiable, establish Taylor’s theorem with remainder

$\displaystyle F(x) = \sum_{j=0}^{k-1} \frac{1}{j!} (((x-x_0) \cdot \nabla)^j F)( x_0 )$

$\displaystyle + \int_0^1 \frac{(1-t)^{k-1}}{(k-1)!} (((x-x_0) \cdot \nabla)^k F)((1-t) x_0 + t x)\ dt.$

If ${\nabla^k F}$ is bounded, conclude that

$\displaystyle |F(x) - \sum_{j=0}^{k-1} \frac{1}{j!} (((x-x_0) \cdot \nabla)^j F)( x_0 )|$

$\displaystyle \leq \frac{|x-x_0|^k}{k!} \sup_{y \in {\bf R}^d} |\nabla^k F(y)|.$

For real scalar functions ${F: {\bf R}^d \rightarrow {\bf R}}$, the average value of the continuous real-valued function ${(x - x_0) \cdot \nabla F((1-t) x_0 + t x)}$ must be attained at some point ${t}$ in the interval ${[0,1]}$. We thus conclude the mean-value theorem

$\displaystyle F(x) - F(x_0) = ((x - x_0) \cdot \nabla F)((1-t) x_0 + t x)$

for some ${t \in [0,1]}$ (that can depend on ${x}$, ${x_0}$, and ${F}$). This can for instance give a second proof of fact that continuously differentiable functions ${F}$ with bounded derivative are Lipschitz continuous. However it is worth stressing that the mean-value theorem is only available for real scalar functions; it is false for instance for complex scalar functions. A basic counterexample is given by the function ${e(x) := e^{2\pi i x}}$; there is no ${t \in [0,1]}$ for which ${e(1) - e(0) = e'(t)}$. On the other hand, as ${e'}$ has magnitude ${2\pi}$, we still know from (4) that ${e}$ is Lipschitz of constant ${2\pi}$, and when combined with (1) we obtain the basic bounds

$\displaystyle |e(x) - e(y)| \leq \min( 2, 2\pi |x-y| )$

which are already very useful for many applications.

Exercise 4 Let ${H_0, V}$ be ${n \times n}$ matrices, and let ${t}$ be a non-negative real.

• (i) Establish the Duhamel formula

$\displaystyle e^{t(H_0+V)} = e^{tH_0} + \int_0^t e^{(t-s) H_0} V e^{s (H_0+V)}\ ds$

$\displaystyle = e^{tH_0} + \int_0^t e^{(t-s) (H_0+V)} V e^{s H_0}\ ds$

where ${e^A}$ denotes the matrix exponential of ${A}$. (Hint: Differentiate ${e^{(t-s) H_0} e^{s (H_0+V)}}$ or ${e^{(t-s) (H_0+V)} e^{s H_0}}$ in ${s}$.)

• (ii) Establish the iterated Duhamel formula

$\displaystyle e^{t(H_0+V)} = e^{tH_0} + \sum_{j=1}^k \int_{0 \leq t_1 \leq \dots \leq t_j \leq t}$

$\displaystyle e^{(t-t_j) H_0} V e^{(t_j-t_{j-1}) H_0} V \dots e^{(t_2-t_1) H_0} V e^{t_1 H_0}\ dt_1 \dots dt_j$

$\displaystyle + \int_{0 \leq t_1 \leq \dots \leq t_{k+1} \leq t}$

$\displaystyle e^{(t-t_{k+1}) (H_0+V)} V e^{(t_{k+1}-t_k) H_0} V \dots e^{(t_2-t_1) H_0} V e^{t_1 H_0}\ dt_1 \dots dt_{k+1}$

for any ${k \geq 0}$.

• (iii) Establish the infinitely iterated Duhamel formula

$\displaystyle e^{t(H_0+V)} = e^{tH_0} + \sum_{j=1}^\infty \int_{0 \leq t_1 \leq \dots \leq t_j \leq t}$

$\displaystyle e^{(t-t_j) H_0} V e^{(t_j-t_{j-1}) H_0} V \dots e^{(t_2-t_1) H_0} V e^{t_1 H_0}\ dt_1 \dots dt_j.$

• (iv) If ${H(t)}$ is an ${n \times n}$ matrix depending in a continuously differentiable fashion on ${t}$, establish the variation formula

$\displaystyle \frac{d}{dt} e^{H(t)} = (F(\mathrm{ad}(H(t))) H'(t)) e^{H(t)}$

where ${\mathrm{ad}(H)}$ is the adjoint representation ${\mathrm{ad}(H)(V) = HV - VH}$ applied to ${H}$, and ${F}$ is the function

$\displaystyle F(z) := \int_0^1 e^{sz}\ ds$

(thus ${F(z) = \frac{e^z-1}{z}}$ for non-zero ${z}$), with ${F(\mathrm{ad}(H(t)))}$ defined using functional calculus.

We remark that further manipulation of (iv) of the above exercise using the fundamental theorem of calculus eventually leads to the Baker-Campbell-Hausdorff-Dynkin formula, as discussed in this previous blog post.

Exercise 5 Let ${A, B}$ be positive definite ${n \times n}$ matrices, and let ${Y}$ be an ${n \times n}$ matrix. Show that there is a unique solution ${X}$ to the Sylvester equation

$\displaystyle AX + X B = Y$

which is given by the formula

$\displaystyle X = \int_0^\infty e^{-tA} Y e^{-tB}\ dt.$

In the above examples we had applied the fundamental theorem of calculus along linear curves ${\gamma(t) = (1-t) x_0 + t x}$. However, it is sometimes better to use other curves. For instance, the circular arc ${\gamma(t) = \cos(\pi t/2) x_0 + \sin(\pi t/2) x}$ can be useful, particularly if ${x_0}$ and ${x}$ are “orthogonal” or “independent” in some sense; a good example of this is the proof by Maurey and Pisier of the gaussian concentration inequality, given in Theorem 8 of this previous blog post. In a similar vein, if one wishes to compare a scalar random variable ${X}$ of mean zero and variance one with a Gaussian random variable ${G}$ of mean zero and variance one, it can be useful to introduce the intermediate random variables ${\gamma(t) := (1-t)^{1/2} X + t^{1/2} G}$ (where ${X}$ and ${G}$ are independent); note that these variables have mean zero and variance one, and after coupling them together appropriately they evolve by the Ornstein-Uhlenbeck process, which has many useful properties. For instance, one can use these ideas to establish monotonicity formulae for entropy; see e.g. this paper of Courtade for an example of this and further references. More generally, one can exploit curves ${\gamma}$ that flow according to some geometrically natural ODE or PDE; several examples of this occur famously in Perelman’s proof of the Poincaré conjecture via Ricci flow, discussed for instance in this previous set of lecture notes.

In some cases, it is difficult to compute ${F(x)-F(x_0)}$ or the derivative ${\nabla F}$ directly, but one can instead proceed by implicit differentiation, or some variant thereof. Consider for instance the matrix inversion map ${F(A) := A^{-1}}$ (defined on the open dense subset of ${n \times n}$ matrices consisting of invertible matrices). If one wants to compute ${F(B)-F(A)}$ for ${B}$ close to ${A}$, one can write temporarily write ${F(B) - F(A) = E}$, thus

$\displaystyle B^{-1} - A^{-1} = E.$

Multiplying both sides on the left by ${B}$ to eliminate the ${B^{-1}}$ term, and on the right by ${A}$ to eliminate the ${A^{-1}}$ term, one obtains

$\displaystyle A - B = B E A$

and thus on reversing these steps we arrive at the basic identity

$\displaystyle B^{-1} - A^{-1} = B^{-1} (A - B) A^{-1}. \ \ \ \ \ (6)$

For instance, if ${H_0, V}$ are ${n \times n}$ matrices, and we consider the resolvents

$\displaystyle R_0(z) := (H_0 - z I)^{-1}; \quad R_V(z) := (H_0 + V - zI)^{-1}$

then we have the resolvent identity

$\displaystyle R_V(z) - R_0(z) = - R_V(z) V R_0(z) \ \ \ \ \ (7)$

as long as ${z}$ does not lie in the spectrum of ${H_0}$ or ${H_0+V}$ (for instance, if ${H_0}$, ${V}$ are self-adjoint then one can take ${z}$ to be any strictly complex number). One can iterate this identity to obtain

$\displaystyle R_V(z) = \sum_{j=0}^k (-R_0(z) V)^j R_0(z) + (-R_V(z) V) (-R_0(z) V)^k R_0(z)$

for any natural number ${k}$; in particular, if ${R_0(z) V}$ has operator norm less than one, one has the Neumann series

$\displaystyle R_V(z) = \sum_{j=0}^\infty (-R_0(z) V)^j R_0(z).$

Similarly, if ${A(t)}$ is a family of invertible matrices that depends in a continuously differentiable fashion on a time variable ${t}$, then by implicitly differentiating the identity

$\displaystyle A(t) A(t)^{-1} = I$

in ${t}$ using the product rule, we obtain

$\displaystyle (\frac{d}{dt} A(t)) A(t)^{-1} + A(t) \frac{d}{dt} A(t)^{-1} = 0$

and hence

$\displaystyle \frac{d}{dt} A(t)^{-1} = - A(t)^{-1} (\frac{d}{dt} A(t)) A(t)^{-1}$

(this identity may also be easily derived from (6)). One can then use the fundamental theorem of calculus to obtain variants of (6), for instance by using the curve ${\gamma(t) = (1-t) A + tB}$ we arrive at

$\displaystyle B^{-1} - A^{-1} = \int_0^1 ((1-t)A + tB)^{-1} (A-B) ((1-t)A + tB)^{-1}\ dt$

assuming that the curve stays entirely within the set of invertible matrices. While this identity may seem more complicated than (6), it is more symmetric, which conveys some advantages. For instance, using this identity it is easy to see that if ${A, B}$ are positive definite with ${A>B}$ in the sense of positive definite matrices (that is, ${A-B}$ is positive definite), then ${B^{-1} > A^{-1}}$. (Try to prove this using (6) instead!)

Exercise 6 If ${A}$ is an invertible ${n \times n}$ matrix and ${u, v}$ are ${n \times 1}$ vectors, establish the Sherman-Morrison formula

$\displaystyle (A + t uv^T)^{-1} = A^{-1} - \frac{t}{1 + t v^T A^{-1} u} A^{-1} uv^T A^{-1}$

whenever ${t}$ is a scalar such that ${1 + t v^T A^{-1} u}$ is non-zero. (See also this previous blog post for more discussion of these sorts of identities.)

One can use the Cauchy integral formula to extend these identities to other functions of matrices. For instance, if ${F: {\bf C} \rightarrow {\bf C}}$ is an entire function, and ${\gamma}$ is a counterclockwise contour that goes around the spectrum of both ${H_0}$ and ${H_0+V}$, then we have

$\displaystyle F(H_0+V) = \frac{-1}{2\pi i} \int_\gamma F(z) R_V(z)\ dz$

and similarly

$\displaystyle F(H_0) = \frac{-1}{2\pi i} \int_\gamma F(z) R_0(z)\ dz$

and hence by (7) one has

$\displaystyle F(H_0+V) - F(H_0) = \frac{1}{2\pi i} \int_\gamma F(z) R_V(z) V F_0(z)\ dz;$

similarly, if ${H(t)}$ depends on ${t}$ in a continuously differentiable fashion, then

$\displaystyle \frac{d}{dt} F(H(t)) = \frac{1}{2\pi i} \int_\gamma F(z) (H(t) - zI)^{-1} H'(t) (z) (H(t)-zI)^{-1}\ dz$

as long as ${\gamma}$ goes around the spectrum of ${H(t)}$.

Exercise 7 If ${H(t)}$ is an ${n \times n}$ matrix depending continuously differentiably on ${t}$, and ${F: {\bf C} \rightarrow {\bf C}}$ is an entire function, establish the tracial chain rule

$\displaystyle \frac{d}{dt} \hbox{tr} F(H(t)) = \hbox{tr}(F'(H(t)) H'(t)).$

In a similar vein, given that the logarithm function is the antiderivative of the reciprocal, one can express the matrix logarithm ${\log A}$ of a positive definite matrix by the fundamental theorem of calculus identity

$\displaystyle \log A = \int_0^\infty (I + sI)^{-1} - (A + sI)^{-1}\ ds$

(with the constant term ${(I+tI)^{-1}}$ needed to prevent a logarithmic divergence in the integral). Differentiating, we see that if ${A(t)}$ is a family of positive definite matrices depending continuously on ${t}$, that

$\displaystyle \frac{d}{dt} \log A(t) = \int_0^\infty (A(t) + sI)^{-1} A'(t) (A(t)+sI)^{-1}\ dt.$

This can be used for instance to show that ${\log}$ is a monotone increasing function, in the sense that ${\log A> \log B}$ whenever ${A > B > 0}$ in the sense of positive definite matrices. One can of course integrate this formula to obtain some formulae for the difference ${\log A - \log B}$ of the logarithm of two positive definite matrices ${A,B}$.

To compare the square root ${A^{1/2} - B^{1/2}}$ of two positive definite matrices ${A,B}$ is trickier; there are multiple ways to proceed. One approach is to use contour integration as before (but one has to take some care to avoid branch cuts of the square root). Another to express the square root in terms of exponentials via the formula

$\displaystyle A^{1/2} = \frac{1}{\Gamma(-1/2)} \int_0^\infty (e^{-tA} - I) t^{-1/2} \frac{dt}{t}$

where ${\Gamma}$ is the gamma function; this formula can be verified by first diagonalising ${A}$ to reduce to the scalar case and using the definition of the Gamma function. Then one has

$\displaystyle A^{1/2} - B^{1/2} = \frac{1}{\Gamma(-1/2)} \int_0^\infty (e^{-tA} - e^{-tB}) t^{-1/2} \frac{dt}{t}$

and one can use some of the previous identities to control ${e^{-tA} - e^{-tB}}$. This is pretty messy though. A third way to proceed is via implicit differentiation. If for instance ${A(t)}$ is a family of positive definite matrices depending continuously differentiably on ${t}$, we can differentiate the identity

$\displaystyle A(t)^{1/2} A(t)^{1/2} = A(t)$

to obtain

$\displaystyle A(t)^{1/2} \frac{d}{dt} A(t)^{1/2} + (\frac{d}{dt} A(t)^{1/2}) A(t)^{1/2} = \frac{d}{dt} A(t).$

This can for instance be solved using Exercise 5 to obtain

$\displaystyle \frac{d}{dt} A(t)^{1/2} = \int_0^\infty e^{-sA(t)^{1/2}} A'(t) e^{-sA(t)^{1/2}}\ ds$

and this can in turn be integrated to obtain a formula for ${A^{1/2} - B^{1/2}}$. This is again a rather messy formula, but it does at least demonstrate that the square root is a monotone increasing function on positive definite matrices: ${A > B > 0}$ implies ${A^{1/2} > B^{1/2} > 0}$.

Several of the above identities for matrices can be (carefully) extended to operators on Hilbert spaces provided that they are sufficiently well behaved (in particular, if they have a good functional calculus, and if various spectral hypotheses are obeyed). We will not attempt to do so here, however.

Suppose one has a bounded sequence ${(a_n)_{n=1}^\infty = (a_1, a_2, \dots)}$ of real numbers. What kinds of limits can one form from this sequence?

Of course, we have the usual notion of limit ${\lim_{n \rightarrow \infty} a_n}$, which in this post I will refer to as the classical limit to distinguish from the other limits discussed in this post. The classical limit, if it exists, is the unique real number ${L}$ such that for every ${\varepsilon>0}$, one has ${|a_n-L| \leq \varepsilon}$ for all sufficiently large ${n}$. We say that a sequence is (classically) convergent if its classical limit exists. The classical limit obeys many useful limit laws when applied to classically convergent sequences. Firstly, it is linear: if ${(a_n)_{n=1}^\infty}$ and ${(b_n)_{n=1}^\infty}$ are classically convergent sequences, then ${(a_n+b_n)_{n=1}^\infty}$ is also classically convergent with

$\displaystyle \lim_{n \rightarrow \infty} (a_n + b_n) = (\lim_{n \rightarrow \infty} a_n) + (\lim_{n \rightarrow \infty} b_n) \ \ \ \ \ (1)$

and similarly for any scalar ${c}$, ${(ca_n)_{n=1}^\infty}$ is classically convergent with

$\displaystyle \lim_{n \rightarrow \infty} (ca_n) = c \lim_{n \rightarrow \infty} a_n. \ \ \ \ \ (2)$

It is also an algebra homomorphism: ${(a_n b_n)_{n=1}^\infty}$ is also classically convergent with

$\displaystyle \lim_{n \rightarrow \infty} (a_n b_n) = (\lim_{n \rightarrow \infty} a_n) (\lim_{n \rightarrow \infty} b_n). \ \ \ \ \ (3)$

We also have shift invariance: if ${(a_n)_{n=1}^\infty}$ is classically convergent, then so is ${(a_{n+1})_{n=1}^\infty}$ with

$\displaystyle \lim_{n \rightarrow \infty} a_{n+1} = \lim_{n \rightarrow \infty} a_n \ \ \ \ \ (4)$

and more generally in fact for any injection ${\phi: {\bf N} \rightarrow {\bf N}}$, ${(a_{\phi(n)})_{n=1}^\infty}$ is classically convergent with

$\displaystyle \lim_{n \rightarrow \infty} a_{\phi(n)} = \lim_{n \rightarrow \infty} a_n. \ \ \ \ \ (5)$

The classical limit of a sequence is unchanged if one modifies any finite number of elements of the sequence. Finally, we have boundedness: for any classically convergent sequence ${(a_n)_{n=1}^\infty}$, one has

$\displaystyle \inf_n a_n \leq \lim_{n \rightarrow \infty} a_n \leq \sup_n a_n. \ \ \ \ \ (6)$

One can in fact show without much difficulty that these laws uniquely determine the classical limit functional on convergent sequences.

One would like to extend the classical limit notion to more general bounded sequences; however, when doing so one must give up one or more of the desirable limit laws that were listed above. Consider for instance the sequence ${a_n = (-1)^n}$. On the one hand, one has ${a_n^2 = 1}$ for all ${n}$, so if one wishes to retain the homomorphism property (3), any “limit” of this sequence ${a_n}$ would have to necessarily square to ${1}$, that is to say it must equal ${+1}$ or ${-1}$. On the other hand, if one wished to retain the shift invariance property (4) as well as the homogeneity property (2), any “limit” of this sequence would have to equal its own negation and thus be zero.

Nevertheless there are a number of useful generalisations and variants of the classical limit concept for non-convergent sequences that obey a significant portion of the above limit laws. For instance, we have the limit superior

$\displaystyle \limsup_{n \rightarrow \infty} a_n := \inf_N \sup_{n \geq N} a_n$

$\displaystyle \liminf_{n \rightarrow \infty} a_n := \sup_N \inf_{n \geq N} a_n$

which are well-defined real numbers for any bounded sequence ${(a_n)_{n=1}^\infty}$; they agree with the classical limit when the sequence is convergent, but disagree otherwise. They enjoy the shift-invariance property (4), and the boundedness property (6), but do not in general obey the homomorphism property (3) or the linearity property (1); indeed, we only have the subadditivity property

$\displaystyle \limsup_{n \rightarrow \infty} (a_n + b_n) \leq (\limsup_{n \rightarrow \infty} a_n) + (\limsup_{n \rightarrow \infty} b_n)$

for the limit superior, and the superadditivity property

$\displaystyle \liminf_{n \rightarrow \infty} (a_n + b_n) \geq (\liminf_{n \rightarrow \infty} a_n) + (\liminf_{n \rightarrow \infty} b_n)$

for the limit inferior. The homogeneity property (2) is only obeyed by the limits superior and inferior for non-negative ${c}$; for negative ${c}$, one must have the limit inferior on one side of (2) and the limit superior on the other, thus for instance

$\displaystyle \limsup_{n \rightarrow \infty} (-a_n) = - \liminf_{n \rightarrow \infty} a_n.$

The limit superior and limit inferior are examples of limit points of the sequence, which can for instance be defined as points that are limits of at least one subsequence of the original sequence. Indeed, the limit superior is always the largest limit point of the sequence, and the limit inferior is always the smallest limit point. However, limit points can be highly non-unique (indeed they are unique if and only if the sequence is classically convergent), and so it is difficult to sensibly interpret most of the usual limit laws in this setting, with the exception of the homogeneity property (2) and the boundedness property (6) that are easy to state for limit points.

Another notion of limit are the Césaro limits

$\displaystyle \mathrm{C}\!-\!\lim_{n \rightarrow \infty} a_n := \lim_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N a_n;$

if this limit exists, we say that the sequence is Césaro convergent. If the sequence ${(a_n)_{n=1}^\infty}$ already has a classical limit, then it also has a Césaro limit that agrees with the classical limit; but there are additional sequences that have a Césaro limit but not a classical one. For instance, the non-classically convergent sequence ${a_n= (-1)^n}$ discussed above is Césaro convergent, with a Césaro limit of ${0}$. However, there are still bounded sequences that do not have Césaro limit, such as ${a_n := \sin( \log n )}$ (exercise!). The Césaro limit is linear, bounded, and shift invariant, but not an algebra homomorphism and also does not obey the rearrangement property (5).

Using the Hahn-Banach theorem, one can extend the classical limit functional to generalised limit functionals ${\mathop{\widetilde \lim}_{n \rightarrow \infty} a_n}$, defined to be bounded linear functionals from the space ${\ell^\infty({\bf N})}$ of bounded real sequences to the real numbers ${{\bf R}}$ that extend the classical limit functional (defined on the space ${c_0({\bf N}) + {\bf R}}$ of convergent sequences) without any increase in the operator norm. (In some of my past writings I made the slight error of referring to these generalised limit functionals as Banach limits, though as discussed below, the latter actually refers to a subclass of generalised limit functionals.) It is not difficult to see that such generalised limit functionals will range between the limit inferior and limit superior. In fact, for any specific sequence ${(a_n)_{n=1}^\infty}$ and any number ${L}$ lying in the closed interval ${[\liminf_{n \rightarrow \infty} a_n, \limsup_{n \rightarrow \infty} a_n]}$, there exists at least one generalised limit functional ${\mathop{\widetilde \lim}_{n \rightarrow \infty}}$ that takes the value ${L}$ when applied to ${a_n}$; for instance, for any number ${\theta}$ in ${[-1,1]}$, there exists a generalised limit functional that assigns that number ${\theta}$ as the “limit” of the sequence ${a_n = (-1)^n}$. This claim can be seen by first designing such a limit functional on the vector space spanned by the convergent sequences and by ${(a_n)_{n=1}^\infty}$, and then appealing to the Hahn-Banach theorem to extend to all sequences. This observation also gives a necessary and sufficient criterion for a bounded sequence ${(a_n)_{n=1}^\infty}$ to classically converge to a limit ${L}$, namely that all generalised limits of this sequence must equal ${L}$.

Because of the reliance on the Hahn-Banach theorem, the existence of generalised limits requires the axiom of choice (or some weakened version thereof); there are presumably models of set theory without the axiom of choice in which no generalised limits exist, but I do not know of an explicit reference for this.

Generalised limits can obey the shift-invariance property (4) or the algebra homomorphism property (2), but as the above analysis of the sequence ${a_n = (-1)^n}$ shows, they cannot do both. Generalised limits that obey the shift-invariance property (4) are known as Banach limits; one can for instance construct them by applying the Hahn-Banach theorem to the Césaro limit functional; alternatively, if ${\mathop{\widetilde \lim}}$ is any generalised limit, then the Césaro-type functional ${(a_n)_{n=1}^\infty \mapsto \mathop{\widetilde \lim}_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N a_n}$ will be a Banach limit. The existence of Banach limits can be viewed as a demonstration of the amenability of the natural numbers (or integers); see this previous blog post for further discussion.

Generalised limits that obey the algebra homomorphism property (2) are known as ultrafilter limits. If one is given a generalised limit functional ${p\!-\!\lim_{n \rightarrow \infty}}$ that obeys (2), then for any subset ${A}$ of the natural numbers ${{\bf N}}$, the generalised limit ${p\!-\!\lim_{n \rightarrow \infty} 1_A(n)}$ must equal its own square (since ${1_A(n)^2 = 1_A(n)}$) and is thus either ${0}$ or ${1}$. If one defines ${p \subset 2^{2^{\bf N}}}$ to be the collection of all subsets ${A}$ of ${{\bf N}}$ for which ${p\!-\!\lim_{n \rightarrow \infty} 1_A(n) = 1}$, one can verify that ${p}$ obeys the axioms of a non-principal ultrafilter. Conversely, if ${p}$ is a non-principal ultrafilter, one can define the associated generalised limit ${p\!-\!\lim_{n \rightarrow \infty} a_n}$ of any bounded sequence ${(a_n)_{n=1}^\infty}$ to be the unique real number ${L}$ such that the sets ${\{ n \in {\bf N}: |a_n - L| \leq \varepsilon \}}$ lie in ${p}$ for all ${\varepsilon>0}$; one can check that this does indeed give a well-defined generalised limit that obeys (2). Non-principal ultrafilters can be constructed using Zorn’s lemma. In fact, they do not quite need the full strength of the axiom of choice; see the Wikipedia article on the ultrafilter lemma for examples.

We have previously noted that generalised limits of a sequence can converge to any point between the limit inferior and limit superior. The same is not true if one restricts to Banach limits or ultrafilter limits. For instance, by the arguments already given, the only possible Banach limit for the sequence ${a_n = (-1)^n}$ is zero. Meanwhile, an ultrafilter limit must converge to a limit point of the original sequence, but conversely every limit point can be attained by at least one ultrafilter limit; we leave these assertions as an exercise to the interested reader. In particular, a bounded sequence converges classically to a limit ${L}$ if and only if all ultrafilter limits converge to ${L}$.

There is no generalisation of the classical limit functional to any space that includes non-classically convergent sequences that obeys the subsequence property (5), since any non-classically-convergent sequence will have one subsequence that converges to the limit superior, and another subsequence that converges to the limit inferior, and one of these will have to violate (5) since the limit superior and limit inferior are distinct. So the above limit notions come close to the best generalisations of limit that one can use in practice.

We summarise the above discussion in the following table:

 Limit Always defined Linear Shift-invariant Homomorphism Constructive Classical No Yes Yes Yes Yes Superior Yes No Yes No Yes Inferior Yes No Yes No Yes Césaro No Yes Yes No Yes Generalised Yes Yes Depends Depends No Banach Yes Yes Yes No No Ultrafilter Yes Yes No Yes No

Ben Green and I have (finally!) uploaded to the arXiv our paper “New bounds for Szemerédi’s theorem, III: A polylogarithmic bound for ${r_4(N)}$“, submitted to Mathematika. This is the sequel to two previous papers (and an erratum to the former paper), concerning quantitative versions of Szemerédi’s theorem in the case of length four progressions. This sequel has been delayed for over a decade for a number of reasons, but we have finally managed to write the arguments up to our satisfaction and submit it (to a special issue of Mathematika honouring the work of Klaus Roth).

For any natural number ${N}$, define ${r_4(N)}$ to be the largest cardinality of a subset ${A}$ of ${[N] = \{1,\dots,N\}}$ which does not contain any non-trivial arithmetic progressions ${a, a+r, a+2r, a+3r}$ of length four (where “non-trivial” means that ${r}$ is non-zero). Trivially we have ${r_4(N) \leq N}$. In 1969, Szemerédi showed that ${r_4(N) = o(N)}$. However, the decay rate that could be theoretically extracted from this argument (and from several subsequent proofs of this bound, including one by Roth) were quite poor. The first significant quantitative bound on this quantity was by Gowers, who showed that ${r_4(N) \ll N (\log \log N)^{-c}}$ for some absolute constant ${c>0}$. In the second paper in the above-mentioned series, we managed to improve this bound to ${r_4(N) \ll N \exp( - c \sqrt{\log \log N})}$. In this paper, we improve the bound further to ${r_4(N) \ll N (\log N)^{-c}}$, which seems to be the limit of the methods. (We remark that if we could take ${c}$ to be larger than one, this would imply the length four case of a well known conjecture of Erdös that any set of natural numbers whose sum of reciprocals diverges would contain arbitrarily long arithmetic progressions. Thanks to the work of Sanders and of Bloom, the corresponding case of the conjecture for length three conjectures is nearly settled, as it is known that for the analogous bound on ${r_3(N)}$ one can take any ${c}$ less than one.)

Most of the previous work on bounding ${r_4(N)}$ relied in some form or another on the density increment argument introduced by Roth back in 1953; roughly speaking, the idea is to show that if a dense subset ${A}$ of ${[N]}$ fails to contain arithmetic progressions of length four, one seeks to then locate a long subprogression of ${[N]}$ in which ${A}$ has increased density. This was the basic method for instance underlying our previous bound ${r_4(N) \ll N \exp( - c \sqrt{\log \log N})}$, as well as a finite field analogue of the bound ${r_4(N) \ll N (\log N)^{-c}}$; however we encountered significant technical difficulties for several years in extending this argument to obtain the result of the current paper. Our method is instead based on “energy increment arguments”, and more specifically on establishing quantitative version of a Khintchine-type recurrence theorem, similar to the qualitative recurrence theorems established (in the ergodic theory context) by Bergelson-Host-Kra, and (in the current combinatorial context) by Ben Green and myself.

One way to phrase the latter recurrence theorem is as follows. Suppose that ${A \subset [N]}$ has density ${\delta}$. Then one would expect a “randomly” selected arithmetic progression ${{\bf a}, {\bf a}+{\bf r}, {\bf a}+2{\bf r}, {\bf a}+3{\bf r}}$ in ${[N]}$ (using the convention that random variables will be in boldface) to be contained in ${A}$ with probability about ${\delta^4}$. This is not true in general, however it was shown by Ben and myself that for any ${\eta>0}$, there was a set of shifts ${r \in [-N,N]}$ of cardinality ${\gg_{\delta,\eta} N}$, such that for any such ${r}$ one had

$\displaystyle {\bf P}( {\bf a}, {\bf a}+r, {\bf a}+2r, {\bf a}+3r \in A ) \geq \delta^4 - \eta$

if ${{\bf a}}$ was chosen uniformly at random from ${[N]}$. This easily implies that ${r_4(N) = o(N)}$, but does not give a particularly good bound on the decay rate, because the implied constant in the cardinality lower bound ${\gg_{\delta,\eta} N}$ is quite poor (in fact of tower-exponential type, due to the use of regularity lemmas!), and so one has to take ${N}$ to be extremely large compared to ${\delta,\eta}$ to avoid the possibility that the set of shifts in the above theorem consists only of the trivial shift ${r=0}$.

We do not know how to improve the lower bound on the set of shifts to the point where it can give bounds that are competitive with those in this paper. However, we can obtain better quantitative results if we permit ourselves to couple together the two parameters ${{\bf a}}$ and ${{\bf r}}$ of the length four progression. Namely, with ${A}$, ${\delta}$, ${\eta}$ as above, we are able to show that there exist random variables ${{\bf a}, {\bf r}}$, not necessarily independent, such that

$\displaystyle {\bf P}( {\bf a}, {\bf a}+{\bf r}, {\bf a}+2{\bf r}, {\bf a}+3{\bf r} \in A ) \geq \delta^4 - \eta \ \ \ \ \ (1)$

and such that we have the non-degeneracy bound

$\displaystyle {\bf P}( {\bf r} = 0 ) \ll \exp( - \eta^{-O(1)} ) / N.$

This then easily implies the main theorem.

The energy increment method is then deployed to locate a good pair ${({\bf a}, {\bf r})}$ of random variables that will obey the above bounds. One can get some intuition on how to proceed here by considering some model cases. Firstly one can consider a “globally quadratically structured” case in which the indicator function ${1_A}$ “behaves like” a globally quadratic function such as ${F( \alpha n^2 )}$, for some irrational ${\alpha}$ and some smooth periodic function ${F: {\bf R}/{\bf Z} \rightarrow {\bf R}}$ of mean ${\delta}$. If one then takes ${{\bf a}, {\bf r}}$ to be uniformly distributed in ${[N]}$ and ${[-\varepsilon N, \varepsilon N]}$ respectively for some small ${\varepsilon>0}$, with no coupling between the two variables, then the left-hand side of (1) is approximately of the form

$\displaystyle \int_{(x,y,z,w) \in ({\bf R}/{\bf Z})^4: x-3y+3z-w = 0} F(x) F(y) F(z) F(w) \ \ \ \ \ (2)$

where the integral is with respect to the probability Haar measure, and the constraint ${x-3y+3z-w=0}$ ultimately arises from the algebraic constraint

$\displaystyle \alpha {\bf a}^2 - 3 \alpha ({\bf a}+{\bf r})^2 + 3 \alpha ({\bf a}+2{\bf r})^2 - \alpha ({\bf a}+3{\bf r})^2 = 0.$

However, an application of the Cauchy-Schwarz inequality and Fubini’s theorem shows that the integral in (2) is at least ${(\int_{{\bf R}/{\bf Z}} F)^4}$, which (morally at least) gives (1) in this case.

Due to the nature of the energy increment argument, it also becomes necessary to consider “locally quadratically structured” cases, in which ${[N]}$ is partitioned into some number of structured pieces ${B_c}$ (think of these as arithmetic progressions, or as “Bohr sets), and on each piece ${B_c}$, ${1_A}$ behaves like a locally quadratic function such as ${F_c( \alpha_c n^2 )}$, where ${\alpha_c}$ now varies with ${c}$, and the mean of ${F_c}$ will be approximately ${\delta}$ on the average after averaging in ${c}$ (weighted by the size of the pieces ${B_c}$). Now one should select ${{\bf a}}$ and ${{\bf r}}$ in the following coupled manner: first one chooses ${{\bf a}}$ uniformly from ${[N]}$, then one defines ${{\bf c}}$ to be the label ${c}$ such that ${{\bf a} \in B_c}$, and then selects ${{\bf r}}$ uniformly from a set ${B_{c,\varepsilon}}$ which is related to ${B_c}$ in much the same way that ${[-\varepsilon N, \varepsilon N]}$ is related to ${[N]}$. If one does this correctly, the analogue of (2) becomes

$\displaystyle {\bf E} \int_{(x,y,z,w) \in ({\bf R}/{\bf Z})^4: x-3y+3z-w = 0} F_{\mathbf c}(x) F_{\mathbf c}(y) F_{\mathbf c}(z) F_{\mathbf c}(w),$

and one can again use Cauchy-Schwarz and Fubini’s theorem to conclude.

The general case proceeds, very roughly, by an iterative argument. At each stage of the iteration, one has some sort of quadratic model of ${1_A}$ which involves a decomposition of ${[N]}$ into structured pieces ${B_c}$, and a quadratic approximation to ${1_A}$ on each piece. If this approximation is accurate enough (or more precisely, if a certain (averaged) local Gowers uniformity norm ${U^3}$ of the error is small enough) to model the count in (1) (for random variables ${{\bf a}, {\bf r}}$ determined by the above partition of ${[N]}$ into pieces ${B_c}$), and if the frequencies (such as ${\alpha_c}$) involved in the quadratic approximation are “high rank” or “linearly independent over the rationals” in a suitably quantitative sense, then some version of the above arguments can be made to work. If there are some unwanted linear dependencies in the frequencies, we can do some linear algebra to eliminate one of the frequencies (using some geometry of numbers to keep the quantitative bounds under control) and continue the iteration. If instead the approximation is too inaccurate, then the error will be large in a certain averaged local Gowers uniformity norm ${U^3}$. A significant fraction of the paper is then devoted to establishing a quantitative inverse theorem for that norm that concludes (with good bounds) that the error must then locally correlate with locally quadratic phases, which can be used to refine the quadratic approximation to ${1_A}$ in a manner that significantly increases its “energy” (basically an ${L^2}$ norm). Such energy increments cannot continue indefinitely, and when they terminate we obtain the desired claim.

There are existing inverse theorems for ${U^3}$ type norms in the literature, going back to the pioneering work of Gowers mentioned previously, and relying on arithmetic combinatorics tools such as Freiman’s theorem and the Balog-Szemerédi-Gowers lemma, which are good for analysing the “${1\%}$-structured homomorphisms” that arise in Gowers’ argument. However, when we applied these methods to the local Gowers norms we obtained inferior quantitative results that were not strong enough for our application. Instead, we use arguments from a different paper of Gowers in which he tackled Szemerédi’s theorem for arbitrary length progressions. This method produces “${99\%}$-structured homomorphisms” associated to any function with large Gowers uniformity norm; however the catch is that such homomorphisms are initially supported only on a sparse unstructured set, rather than a structured set such as a Bohr set. To proceed further, one first has to locate inside the sparse unstructured set a sparse pseudorandom subset of a Bohr set, and then use “error-correction” type methods (such as “majority-vote” based algorithms) to locally upgrade this ${99\%}$-structured homomorphism on pseudorandom subsets of Bohr sets to a ${100\%}$-structured homomorphism on the entirety of a Bohr set. It is then possible to use some “approximate cohomology” tools to “integrate” these homomorphisms (and discern a key “local symmetry” property of these homomorphisms) to locate the desired local quadratic structure (in much the same fashion that a ${1}$-form on ${{\bf R}^n}$ that varies linearly with the coordinates can be integrated to be the derivative of a quadratic function if we know that the ${1}$-form is closed). These portions of the paper are unfortunately rather technical, but broadly follow the methods already used in previous literature.

A sequence ${a: {\bf Z} \rightarrow {\bf C}}$ of complex numbers is said to be quasiperiodic if it is of the form

$\displaystyle a(n) = F( \alpha_1 n \hbox{ mod } 1, \dots, \alpha_k n \hbox{ mod } 1)$

for some real numbers ${\alpha_1,\dots,\alpha_k}$ and continuous function ${F: ({\bf R}/{\bf Z})^k \rightarrow {\bf C}}$. For instance, linear phases such as ${n \mapsto e(\alpha n + \beta)}$ (where ${e(\theta) := e^{2\pi i \theta}}$) are examples of quasiperiodic sequences; the top order coefficient ${\alpha}$ (modulo ${1}$) can be viewed as a “frequency” of the integers, and an element of the Pontryagin dual ${\hat {\bf Z} \equiv {\bf R}/{\bf Z}}$ of the integers. Any periodic sequence is also quasiperiodic (taking ${k=1}$ and ${\alpha_1}$ to be the reciprocal of the period). A sequence is said to be almost periodic if it is the uniform limit of quasiperiodic sequences. For instance any Fourier series of the form

$\displaystyle a(n) = \sum_{j=1}^\infty c_j e(\alpha_j n)$

with ${\alpha_1,\alpha_2,\dots}$ real numbers and ${c_1,c_2,\dots}$ an absolutely summable sequence of complex coefficients, will be almost periodic.

These sequences arise in various “complexity one” problems in arithmetic combinatorics and ergodic theory. For instance, if ${(X, \mu, T)}$ is a measure-preserving system – a probability space ${(X,\mu)}$ equipped with a measure-preserving shift, and ${f_1,f_2 \in L^\infty(X)}$ are bounded measurable functions, then the correlation sequence

$\displaystyle a(n) := \int_X f_1(x) f_2(T^n x)\ d\mu(x)$

can be shown to be an almost periodic sequence, plus an error term ${b_n}$ which is “null” in the sense that it has vanishing uniform density:

$\displaystyle \sup_N \frac{1}{M} \sum_{n=N+1}^{N+M} |b_n| \rightarrow 0 \hbox{ as } M \rightarrow \infty.$

This can be established in a number of ways, for instance by writing ${a(n)}$ as the Fourier coefficients of the spectral measure of the shift ${T}$ with respect to the functions ${f_1,f_2}$, and then decomposing that measure into pure point and continuous components.

In the last two decades or so, it has become clear that there are natural higher order versions of these concepts, in which linear polynomials such as ${\alpha n + \beta}$ are replaced with higher degree counterparts. The most obvious candidates for these counterparts would be the polynomials ${\alpha_d n^d + \dots + \alpha_0}$, but this turns out to not be a complete set of higher degree objects needed for the theory. Instead, the higher order versions of quasiperiodic and almost periodic sequences are now known as basic nilsequences and nilsequences respectively, while the higher order version of a linear phase is a nilcharacter; each nilcharacter then has a symbol that is a higher order generalisation of the concept of a frequency (and the collection of all symbols forms a group that can be viewed as a higher order version of the Pontryagin dual of ${{\bf Z}}$). The theory of these objects is spread out in the literature across a number of papers; in particular, the theory of nilcharacters is mostly developed in Appendix E of this 116-page paper of Ben Green, Tamar Ziegler, and myself, and is furthermore written using nonstandard analysis and treating the more general setting of higher dimensional sequences. I therefore decided to rewrite some of that material in this blog post, in the simpler context of the qualitative asymptotic theory of one-dimensional nilsequences and nilcharacters rather than the quantitative single-scale theory that is needed for combinatorial applications (and which necessitated the use of nonstandard analysis in the previous paper).

For technical reasons (having to do with the non-trivial topological structure on nilmanifolds), it will be convenient to work with vector-valued sequences, that take values in a finite-dimensional complex vector space ${{\bf C}^m}$ rather than in ${{\bf C}}$. By doing so, the space of sequences is now, technically, no longer a ring, as the operations of addition and multiplication on vector-valued sequences become ill-defined. However, we can still take complex conjugates of a sequence, and add sequences taking values in the same vector space ${{\bf C}^m}$, and for sequences taking values in different vector spaces ${{\bf C}^m}$, ${{\bf C}^{m'}}$, we may utilise the tensor product ${\otimes: {\bf C}^m \times {\bf C}^{m'} \rightarrow {\bf C}^{mm'}}$, which we will normalise by defining

$\displaystyle (z_1,\dots,z_m) \otimes (w_1,\dots,w_{m'}) = (z_1 w_1, \dots, z_1 w_{m'}, \dots, z_m w_1, \dots, z_m w_{m'} ).$

This product is associative and bilinear, and also commutative up to permutation of the indices. It also interacts well with the Hermitian norm

$\displaystyle \| (z_1,\dots,z_m) \| := \sqrt{|z_1|^2 + \dots + |z_m|^2}$

since we have ${\|z \otimes w\| = \|z\| \|w\|}$.

The traditional definition of a basic nilsequence (as defined for instance by Bergelson, Host, and Kra) is as follows:

Definition 1 (Basic nilsequence, first definition) A nilmanifold of step at most ${d}$ is a quotient ${G/\Gamma}$, where ${G}$ is a connected, simply connected nilpotent Lie group of step at most ${d}$ (thus, all ${d+1}$-fold commutators vanish) and ${\Gamma}$ is a discrete cocompact lattice in ${G}$. A basic nilsequence of degree at most ${d}$ is a sequence of the form ${n \mapsto F(g^n g_0 \Gamma)}$, where ${g_0 \Gamma \in G/\Gamma}$, ${g \in G}$, and ${F: G/\Gamma \rightarrow {\bf C}^m}$ is a continuous function.

For instance, it is not difficult using this definition to show that a sequence is a basic nilsequence of degree at most ${1}$ if and only if it is quasiperiodic. The requirement that ${G}$ be simply connected can be easily removed if desired by passing to a universal cover, but it is technically convenient to assume it (among other things, it allows for a well-defined logarithm map that obeys the Baker-Campbell-Hausdorff formula). When one wishes to perform a more quantitative analysis of nilsequences (particularly when working on a “single scale”. sich as on a single long interval ${\{ N+1, \dots, N+M\}}$), it is common to impose additional regularity conditions on the function ${F}$, such as Lipschitz continuity or smoothness, but ordinary continuity will suffice for the qualitative discussion in this blog post.

Nowadays, and particularly when one needs to understand the “single-scale” equidistribution properties of nilsequences, it is more convenient (as is for instance done in this ICM paper of Green) to use an alternate definition of a nilsequence as follows.

Definition 2 Let ${d \geq 0}$. A filtered group of degree at most ${d}$ is a group ${G}$ together with a sequence ${G_\bullet = (G_0,G_1,G_2,\dots)}$ of subgroups ${G \geq G_0 \geq G_1 \geq \dots}$ with ${G_{d+1}=\{\hbox{id}\}}$ and ${[G_i,G_j] \subset G_{i+j}}$ for ${i,j \geq 0}$. A polynomial sequence ${g: {\bf Z} \rightarrow G}$ into a filtered group is a function such that ${\partial_{h_i} \dots \partial_{h_1} g(n) \in G_i}$ for all ${i \geq 0}$ and ${n,h_1,\dots,h_i \in{\bf Z}}$, where ${\partial_h g(n) := g(n+h) g(n)^{-1}}$ is the difference operator. A filtered nilmanifold of degree at most ${s}$ is a quotient ${G/\Gamma}$, where ${G}$ is a filtered group of degree at most ${s}$ such that ${G}$ and all of the subgroups ${G_i}$ are connected, simply connected nilpotent filtered Lie group, and ${\Gamma}$ is a discrete cocompact subgroup of ${G}$ such that ${\Gamma_i := \Gamma \cap G_i}$ is a discrete cocompact subgroup of ${G_i}$. A basic nilsequence of degree at most ${d}$ is a sequence of the form ${n \mapsto F(g(n))}$, where ${g: {\bf Z} \rightarrow G}$ is a polynomial sequence, ${G/\Gamma}$ is a filtered nilmanifold of degree at most ${d}$, and ${F: G \rightarrow {\bf C}^m}$ is a continuous function which is ${\Gamma}$-automorphic, in the sense that ${F(g \gamma) = F(g)}$ for all ${g \in G}$ and ${\gamma \in \Gamma}$.

One can easily identify a ${\Gamma}$-automorphic function on ${G}$ with a function on ${G/\Gamma}$, but there are some (very minor) advantages to working on the group ${G}$ instead of the quotient ${G/\Gamma}$, as it becomes slightly easier to modify the automorphy group ${\Gamma}$ when needed. (But because the action of ${\Gamma}$ on ${G}$ is free, one can pass from ${\Gamma}$-automorphic functions on ${G}$ to functions on ${G/\Gamma}$ with very little difficulty.) The main reason to work with polynomial sequences ${n \mapsto g(n)}$ rather than geometric progressions ${n \mapsto g^n g_0 \Gamma}$ is that they form a group, a fact essentially established by by Lazard and Leibman; see Corollary B.4 of this paper of Green, Ziegler, and myself for a proof in the filtered group setting.

It is easy to see that any sequence that is a basic nilsequence of degree at most ${d}$ in the sense of the first definition, is also a basic nilsequence of degree at most ${d}$ in the second definition, since a nilmanifold of degree at most ${d}$ can be filtered using the lower central series, and any linear sequence ${n \mapsto g^n g_0}$ will be a polynomial sequence with respect to that filtration. The converse implication is a little trickier, but still not too hard to show: see Appendix C of this paper of Ben Green, Tamar Ziegler, and myself. There are two key examples of basic nilsequences to keep in mind. The first are the polynomially quasiperiodic sequences

$\displaystyle a(n) = F( P_1(n), \dots, P_k(n) ),$

where ${P_1,\dots,P_k: {\bf Z} \rightarrow {\bf R}}$ are polynomials of degree at most ${d}$, and ${F: {\bf R}^k \rightarrow {\bf C}^m}$ is a ${{\bf Z}^k}$-automorphic (i.e., ${{\bf Z}^k}$-periodic) continuous function. The map ${P: {\bf Z} \rightarrow {\bf R}^k}$ defined by ${P(n) := (P_1(n),\dots,P_k(n))}$ is a polynomial map of degree at most ${d}$, if one filters ${{\bf R}^k}$ by defining ${({\bf R}^k)_i}$ to equal ${{\bf R}^k}$ when ${i \leq d}$, and ${\{0\}}$ for ${i > d}$. The torus ${{\bf R}^k/{\bf Z}^k}$ then becomes a filtered nilmanifold of degree at most ${d}$, and ${a(n)}$ is thus a basic nilsequence of degree at most ${d}$ as per the second definition. It is also possible explicitly describe ${a_n}$ as a basic nilsequence of degree at most ${d}$ as per the first definition, for instance (in the ${k=1}$ case) by taking ${G}$ to be the space of upper triangular unipotent ${d+1 \times d+1}$ real matrices, and ${\Gamma}$ the subgroup with integer coefficients; we leave the details to the interested reader.

The other key example is a sequence of the form

$\displaystyle a(n) = F( \alpha n, \{ \alpha n \} \beta n )$

where ${\alpha,\beta}$ are real numbers, ${\{ \alpha n \} = \alpha n - \lfloor \alpha n \rfloor}$ denotes the fractional part of ${\alpha n}$, and and ${F: {\bf R}^2 \rightarrow {\bf C}^m}$ is a ${{\bf Z}^2}$-automorphic continuous function that vanishes in a neighbourhood of ${{\bf Z} \times {\bf R}}$. To describe this as a nilsequence, we use the nilpotent connected, simply connected degree ${2}$, Heisenberg group

$\displaystyle G := \begin{pmatrix} 1 & {\bf R} & {\bf R} \\ 0 & 1 & {\bf R} \\ 0 & 0 & 1 \end{pmatrix}$

with the lower central series filtration ${G_0=G_1=G}$, ${G_2= [G,G] = \begin{pmatrix} 1 &0 & {\bf R} \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}}$, and ${G_i = \{ \mathrm{id} \}}$ for ${i > 2}$, ${\Gamma}$ to be the discrete compact subgroup

$\displaystyle \Gamma := \begin{pmatrix} 1 & {\bf Z} & {\bf Z} \\ 0 & 1 & {\bf Z} \\ 0 & 0 & 1 \end{pmatrix},$

${g: {\bf Z} \rightarrow G}$ to be the polynomial sequence

$\displaystyle g(n) := \begin{pmatrix} 1 & \beta n & \alpha \beta n^2 \\ 0 & 1 & \alpha n \\ 0 & 0 & 1 \end{pmatrix}$

and ${\tilde F: G \rightarrow {\bf C}^m}$ to be the ${\Gamma}$-automorphic function

$\displaystyle \tilde F( \begin{pmatrix} 1 & x & y \\ 0 & 1 & z \\ 0 & 0 & 1 \end{pmatrix} ) = F( \{ z \}, y - \lfloor z \rfloor x );$

one easily verifies that this function is indeed ${\Gamma}$-automorphic, and it is continuous thanks to the vanishing properties of ${F}$. Also we have ${a(n) = \tilde F(g(n))}$, so ${a}$ is a basic nilsequence of degree at most ${2}$. One can concoct similar examples with ${\{ \alpha n \} \beta n}$ replaced by other “bracket polynomials” of ${n}$; for instance

$\displaystyle a(n) = F( \alpha n, \{ \alpha n - \frac{1}{2} \} \beta n )$

will be a basic nilsequence if ${F}$ now vanishes in a neighbourhood of ${(\frac{1}{2}+{\bf Z}) \times {\bf R}}$ rather than ${{\bf Z} \times {\bf R}}$. See this paper of Bergelson and Leibman for more discussion of bracket polynomials (also known as generalised polynomials) and their relationship to nilsequences.

A nilsequence of degree at most ${d}$ is defined to be a sequence that is the uniform limit of basic nilsequences of degree at most ${d}$. Thus for instance a sequence is a nilsequence of degree at most ${1}$ if and only if it is almost periodic, while a sequence is a nilsequence of degree at most ${0}$ if and only if it is constant. Such objects arise in higher order recurrence: for instance, if ${h_0,\dots,h_d}$ are integers, ${(X,\mu,T)}$ is a measure-preserving system, and ${f_0,\dots,f_d \in L^\infty(X)}$, then it was shown by Leibman that the sequence

$\displaystyle n \mapsto \int_X f_0(T^{h_0 n} x) \dots f_d(T^{h_d n} x)\ d\mu(x)$

is equal to a nilsequence of degree at most ${d}$, plus a null sequence. (The special case when the measure-preserving system was ergodic and ${h_i = i}$ for ${i=0,\dots,d}$ was previously established by Bergelson, Host, and Kra.) Nilsequences also arise in the inverse theory of the Gowers uniformity norms, as discussed for instance in this previous post.

It is easy to see that a sequence ${a: {\bf Z} \rightarrow {\bf C}^m}$ is a basic nilsequence of degree at most ${d}$ if and only if each of its ${m}$ components are. The scalar basic nilsequences ${a: {\bf Z} \rightarrow {\bf C}}$ of degree ${d}$ are easily seen to form a ${*}$-algebra (that is to say, they are a complex vector space closed under pointwise multiplication and complex conjugation), which implies similarly that vector-valued basic nilsequences ${a: {\bf Z} \rightarrow {\bf C}^m}$ of degree at most ${d}$ form a complex vector space closed under complex conjugation for each ${m}$, and that the tensor product of any two basic nilsequences of degree at most ${d}$ is another basic nilsequence of degree at most ${d}$. Similarly with “basic nilsequence” replaced by “nilsequence” throughout.

Now we turn to the notion of a nilcharacter, as defined in this paper of Ben Green, Tamar Ziegler, and myself:

Definition 3 (Nilcharacters) Let ${d \geq 1}$. A sub-nilcharacter of degree ${d}$ is a basic nilsequence ${\chi: n \mapsto F(g(n))}$ of degree at most ${d}$, such that ${F}$ obeys the additional modulation property

$\displaystyle F( g_d g ) = e( \xi \cdot g_d ) F(g) \ \ \ \ \ (1)$

for all ${g \in G}$ and ${g_d \in G_d}$, where ${\xi: G_d \rightarrow {\bf R}}$ is a continuous homomorphism ${g_d \mapsto \xi \cdot g_d}$. (Note from (1) and ${\Gamma}$-automorphy that unless ${F}$ vanishes identically, ${\xi}$ must map ${\Gamma_d}$ to ${{\bf Z}}$, thus without loss of generality one can view ${\xi}$ as an element of the Pontryagial dual of the torus ${G_d / \Gamma_d}$.) If in addition one has ${\|F(g)\|=1}$ for all ${g \in G}$, we call ${\chi}$ a nilcharacter of degree ${d \geq 1}$.

In the degree one case ${d=1}$, the only sub-nilcharacters are of the form ${\chi(n) = e(\alpha n)}$ for some vector ${c \in {\bf C}^m}$ and ${\alpha \in {\bf R}}$, and this is a nilcharacter if ${c}$ is a unit vector. Similarly, in higher degree, any sequence of the form ${\chi(n) = c e(P(n))}$, where ${c \in {\bf C}^m}$ is a vector and ${P: {\bf Z} \rightarrow {\bf R}}$ is a polynomial of degree at most ${d}$, is a sub-nilcharacter of degree ${d}$, and a character if ${c}$ is a unit vector. A nilsequence of degree at most ${d-1}$ is automatically a sub-nilcharacter of degree ${d}$, and a nilcharacter if it is of magnitude ${1}$. A further example of a nilcharacter is provided by the two-dimensional sequence ${\chi: {\bf Z} \rightarrow {\bf C}^2}$ defined by

$\displaystyle \chi(n) := ( F_0( \alpha n ) e( \{ \alpha n \} \beta n ), F_{1/2}( \alpha n ) e( \{ \alpha n - \frac{1}{2} \} \beta n ) ) \ \ \ \ \ (2)$

where ${F_0, F_{1/2}: {\bf R} \rightarrow {\bf C}}$ are continuous, ${{\bf Z}}$-automorphic functions that vanish on a neighbourhood of ${{\bf Z}}$ and ${\frac{1}{2}+{\bf Z}}$ respectively, and which form a partition of unity in the sense that

$\displaystyle |F_0(x)|^2 + |F_{1/2}(x)|^2 = 1$

for all ${x \in {\bf R}}$. Note that one needs both ${F_0}$ and ${F_{1/2}}$ to be not identically zero in order for all these conditions to be satisfied; it turns out (for topological reasons) that there is no scalar nilcharacter that is “equivalent” to this nilcharacter in a sense to be defined shortly. In some literature, one works exclusively with sub-nilcharacters rather than nilcharacters, however the former space contains zero-divisors, which is a little annoying technically. Nevertheless, both nilcharacters and sub-nilcharacters generate the same set of “symbols” as we shall see later.

We claim that every degree ${d}$ sub-nilcharacter ${f: {\bf Z} \rightarrow {\bf C}^m}$ can be expressed in the form ${f = c \chi}$, where ${\chi: {\bf Z} \rightarrow {\bf C}^{m'}}$ is a degree ${d}$ nilcharacter, and ${c: {\bf C}^{m'} \rightarrow {\bf C}^m}$ is a linear transformation. Indeed, by scaling we may assume ${f(n) = F(g(n))}$ where ${|F| < 1}$ uniformly. Using partitions of unity, one can find further functions ${F_1,\dots,F_m}$ also obeying (1) for the same character ${\xi}$ such that ${|F_1|^2 + \dots + |F_m|^2}$ is non-zero; by dividing out the ${F_1,\dots,F_m}$ by the square root of this quantity, and then multiplying by ${\sqrt{1-|F|^2}}$, we may assume that

$\displaystyle |F|^2 + |F_1|^2 + \dots + |F_m|^2 = 1,$

and then

$\displaystyle \chi(n) := (F(g(n)), F_1(g(n)), \dots, F_m(g(n)))$

becomes a degree ${d}$ nilcharacter that contains ${f(n)}$ amongst its components, giving the claim.

As we shall show below, nilsequences can be approximated uniformly by linear combinations of nilcharacters, in much the same way that quasiperiodic or almost periodic sequences can be approximated uniformly by linear combinations of linear phases. In particular, nilcharacters can be used as “obstructions to uniformity” in the sense of the inverse theory of the Gowers uniformity norms.

The space of degree ${d}$ nilcharacters forms a semigroup under tensor product, with the constant sequence ${1}$ as the identity. One can upgrade this semigroup to an abelian group by quotienting nilcharacters out by equivalence:

Definition 4 Let ${d \geq 1}$. We say that two degree ${d}$ nilcharacters ${\chi: {\bf Z} \rightarrow {\bf C}^m}$, ${\chi': {\bf Z} \rightarrow {\bf C}^{m'}}$ are equivalent if ${\chi \otimes \overline{\chi'}: {\bf Z} \rightarrow {\bf C}^{mm'}}$ is equal (as a sequence) to a basic nilsequence of degree at most ${d-1}$. (We will later show that this is indeed an equivalence relation.) The equivalence class ${[\chi]_{\mathrm{Symb}^d({\bf Z})}}$ of such a nilcharacter will be called the symbol of that nilcharacter (in analogy to the symbol of a differential or pseudodifferential operator), and the collection of such symbols will be denoted ${\mathrm{Symb}^d({\bf Z})}$.

As we shall see below the fold, ${\mathrm{Symb}^d({\bf Z})}$ has the structure of an abelian group, and enjoys some nice “symbol calculus” properties; also, one can view symbols as precisely describing the obstruction to equidistribution for nilsequences. For ${d=1}$, the group is isomorphic to the Ponytragin dual ${\hat {\bf Z} = {\bf R}/{\bf Z}}$ of the integers, and ${\mathrm{Symb}^d({\bf Z})}$ for ${d > 1}$ should be viewed as higher order generalisations of this Pontryagin dual. In principle, this group can be explicitly described for all ${d}$, but the theory rapidly gets complicated as ${d}$ increases (much as the classification of nilpotent Lie groups or Lie algebras of step ${d}$ rapidly gets complicated even for medium-sized ${d}$ such as ${d=3}$ or ${d=4}$). We will give an explicit description of the ${d=2}$ case here. There is however one nice (and non-trivial) feature of ${\mathrm{Symb}^d({\bf Z})}$ for ${d \geq 2}$ – it is not just an abelian group, but is in fact a vector space over the rationals ${{\bf Q}}$!

How many groups of order four are there? Technically, there are an enormous number, so much so, in fact, that the class of groups of order four is not even a set, but merely a proper class. This is because any four objects ${a,b,c,d}$ can be turned into a group ${\{a,b,c,d\}}$ by designating one of the four objects, say ${a}$, to be the group identity, and imposing a suitable multiplication table (and inversion law) on the four elements in a manner that obeys the usual group axioms. Since all sets are themselves objects, the class of four-element groups is thus at least as large as the class of all sets, which by Russell’s paradox is known not to itself be a set (assuming the usual ZFC axioms of set theory).

A much better question is to ask how many groups of order four there are up to isomorphism, counting each isomorphism class of groups exactly once. Now, as one learns in undergraduate group theory classes, the answer is just “two”: the cyclic group ${C_4}$ and the Klein four-group ${C_2 \times C_2}$.

More generally, given a class of objects ${X}$ and some equivalence relation ${\sim}$ on ${X}$ (which one should interpret as describing the property of two objects in ${X}$ being “isomorphic”), one can consider the number ${|X / \sim|}$ of objects in ${X}$ “up to isomorphism”, which is simply the cardinality of the collection ${X/\sim}$ of equivalence classes ${[x]:=\{y\in X:x \sim {}y \}}$ of ${X}$. In the case where ${X}$ is finite, one can express this cardinality by the formula

$\displaystyle |X/\sim| = \sum_{x \in X} \frac{1}{|[x]|}, \ \ \ \ \ (1)$

thus one counts elements in ${X}$, weighted by the reciprocal of the number of objects they are isomorphic to.

Example 1 Let ${X}$ be the five-element set ${\{-2,-1,0,1,2\}}$ of integers between ${-2}$ and ${2}$. Let us say that two elements ${x, y}$ of ${X}$ are isomorphic if they have the same magnitude: ${x \sim y \iff |x| = |y|}$. Then the quotient space ${X/\sim}$ consists of just three equivalence classes: ${\{-2,2\} = [2] = [-2]}$, ${\{-1,1\} = [-1] = [1]}$, and ${\{0\} = [0]}$. Thus there are three objects in ${X}$ up to isomorphism, and the identity (1) is then just

$\displaystyle 3 = \frac{1}{2} + \frac{1}{2} + 1 + \frac{1}{2} + \frac{1}{2}.$

Thus, to count elements in ${X}$ up to equivalence, the elements ${-2,-1,1,2}$ are given a weight of ${1/2}$ because they are each isomorphic to two elements in ${X}$, while the element ${0}$ is given a weight of ${1}$ because it is isomorphic to just one element in ${X}$ (namely, itself).

Given a finite probability set ${X}$, there is also a natural probability distribution on ${X}$, namely the uniform distribution, according to which a random variable ${\mathbf{x} \in X}$ is set equal to any given element ${x}$ of ${X}$ with probability ${\frac{1}{|X|}}$:

$\displaystyle {\bf P}( \mathbf{x} = x ) = \frac{1}{|X|}.$

Given a notion ${\sim}$ of isomorphism on ${X}$, one can then define the random equivalence class ${[\mathbf{x}] \in X/\sim}$ that the random element ${\mathbf{x}}$ belongs to. But if the isomorphism classes are unequal in size, we now encounter a biasing effect: even if ${\mathbf{x}}$ was drawn uniformly from ${X}$, the equivalence class ${[\mathbf{x}]}$ need not be uniformly distributed in ${X/\sim}$. For instance, in the above example, if ${\mathbf{x}}$ was drawn uniformly from ${\{-2,-1,0,1,2\}}$, then the equivalence class ${[\mathbf{x}]}$ will not be uniformly distributed in the three-element space ${X/\sim}$, because it has a ${2/5}$ probability of being equal to the class ${\{-2,2\}}$ or to the class ${\{-1,1\}}$, and only a ${1/5}$ probability of being equal to the class ${\{0\}}$.

However, it is possible to remove this bias by changing the weighting in (1), and thus changing the notion of what cardinality means. To do this, we generalise the previous situation. Instead of considering sets ${X}$ with an equivalence relation ${\sim}$ to capture the notion of isomorphism, we instead consider groupoids, which are sets ${X}$ in which there are allowed to be multiple isomorphisms between elements in ${X}$ (and in particular, there are allowed to be multiple automorphisms from an element to itself). More precisely:

Definition 2 A groupoid is a set (or proper class) ${X}$, together with a (possibly empty) collection ${\mathrm{Iso}(x \rightarrow y)}$ of “isomorphisms” between any pair ${x,y}$ of elements of ${X}$, and a composition map ${f, g \mapsto g \circ f}$ from isomorphisms ${f \in \mathrm{Iso}(x \rightarrow y)}$, ${g \in \mathrm{Iso}(y \rightarrow z)}$ to isomorphisms in ${\mathrm{Iso}(x \rightarrow z)}$ for every ${x,y,z \in X}$, obeying the following group-like axioms:

• (Identity) For every ${x \in X}$, there is an identity isomorphism ${\mathrm{id}_x \in \mathrm{Iso}(x \rightarrow x)}$, such that ${f \circ \mathrm{id}_x = \mathrm{id}_y \circ f = f}$ for all ${f \in \mathrm{Iso}(x \rightarrow y)}$ and ${x,y \in X}$.
• (Associativity) If ${f \in \mathrm{Iso}(x \rightarrow y)}$, ${g \in \mathrm{Iso}(y \rightarrow z)}$, and ${h \in \mathrm{Iso}(z \rightarrow w)}$ for some ${x,y,z,w \in X}$, then ${h \circ (g \circ f) = (h \circ g) \circ f}$.
• (Inverse) If ${f \in \mathrm{Iso}(x \rightarrow y)}$ for some ${x,y \in X}$, then there exists an inverse isomorphism ${f^{-1} \in \mathrm{Iso}(y \rightarrow x)}$ such that ${f \circ f^{-1} = \mathrm{id}_y}$ and ${f^{-1} \circ f = \mathrm{id}_x}$.

We say that two elements ${x,y}$ of a groupoid are isomorphic, and write ${x \sim y}$, if there is at least one isomorphism from ${x}$ to ${y}$.

Example 3 Any category gives a groupoid by taking ${X}$ to be the set (or class) of objects, and ${\mathrm{Iso}(x \rightarrow y)}$ to be the collection of invertible morphisms from ${x}$ to ${y}$. For instance, in the category ${\mathbf{Set}}$ of sets, ${\mathrm{Iso}(x \rightarrow y)}$ would be the collection of bijections from ${x}$ to ${y}$; in the category ${\mathbf{Vec}/k}$ of linear vector spaces over some given base field ${k}$, ${\mathrm{Iso}(x \rightarrow y)}$ would be the collection of invertible linear transformations from ${x}$ to ${y}$; and so forth.

Every set ${X}$ equipped with an equivalence relation ${\sim}$ can be turned into a groupoid by assigning precisely one isomorphism ${\iota_{x \rightarrow y}}$ from ${x}$ to ${y}$ for any pair ${x,y \in X}$ with ${x \sim y}$, and no isomorphisms from ${x}$ to ${y}$ when ${x \not \sim y}$, with the groupoid operations of identity, composition, and inverse defined in the only way possible consistent with the axioms. We will call this the simply connected groupoid associated with this equivalence relation. For instance, with ${X = \{-2,-1,0,1,2\}}$ as above, if we turn ${X}$ into a simply connected groupoid, there will be precisely one isomorphism from ${2}$ to ${-2}$, and also precisely one isomorphism from ${2}$ to ${2}$, but no isomorphisms from ${2}$ to ${-1}$, ${0}$, or ${1}$.

However, one can also form multiply-connected groupoids in which there can be multiple isomorphisms from one element of ${X}$ to another. For instance, one can view ${X = \{-2,-1,0,1,2\}}$ as a space that is acted on by multiplication by the two-element group ${\{-1,+1\}}$. This gives rise to two types of isomorphisms, an identity isomorphism ${(+1)_x}$ from ${x}$ to ${x}$ for each ${x \in X}$, and a negation isomorphism ${(-1)_x}$ from ${x}$ to ${-x}$ for each ${x \in X}$; in particular, there are two automorphisms of ${0}$ (i.e., isomorphisms from ${0}$ to itself), namely ${(+1)_0}$ and ${(-1)_0}$, whereas the other four elements of ${X}$ only have a single automorphism (the identity isomorphism). One defines composition, identity, and inverse in this groupoid in the obvious fashion (using the group law of the two-element group ${\{-1,+1\}}$); for instance, we have ${(-1)_{-2} \circ (-1)_2 = (+1)_2}$.

For a finite multiply-connected groupoid, it turns out that the natural notion of “cardinality” (or as I prefer to call it, “cardinality up to isomorphism”) is given by the variant

$\displaystyle \sum_{x \in X} \frac{1}{|\{ f: f \in \mathrm{Iso}(x \rightarrow y) \hbox{ for some } y\}|}$

of (1). That is to say, in the multiply connected case, the denominator is no longer the number of objects ${y}$ isomorphic to ${x}$, but rather the number of isomorphisms from ${x}$ to other objects ${y}$. Grouping together all summands coming from a single equivalence class ${[x]}$ in ${X/\sim}$, we can also write this expression as

$\displaystyle \sum_{[x] \in X/\sim} \frac{1}{|\mathrm{Aut}(x)|} \ \ \ \ \ (2)$

where ${\mathrm{Aut}(x) := \mathrm{Iso}(x \rightarrow x)}$ is the automorphism group of ${x}$, that is to say the group of isomorphisms from ${x}$ to itself. (Note that if ${x,x'}$ belong to the same equivalence class ${[x]}$, then the two groups ${\mathrm{Aut}(x)}$ and ${\mathrm{Aut}(x')}$ will be isomorphic and thus have the same cardinality, and so the above expression is well-defined.

For instance, if we take ${X}$ to be the simply connected groupoid on ${\{-2,-1,0,1,2\}}$, then the number of elements of ${X}$ up to isomorphism is

$\displaystyle \frac{1}{2} + \frac{1}{2} + 1 + \frac{1}{2} + \frac{1}{2} = 1 + 1 + 1 = 3$

exactly as before. If however we take the multiply connected groupoid on ${\{-2,-1,0,1,2\}}$, in which ${0}$ has two automorphisms, the number of elements of ${X}$ up to isomorphism is now the smaller quantity

$\displaystyle \frac{1}{2} + \frac{1}{2} + \frac{1}{2} + \frac{1}{2} + \frac{1}{2} = 1 + \frac{1}{2} + 1 = \frac{5}{2};$

the equivalence class ${[0]}$ is now counted with weight ${1/2}$ rather than ${1}$ due to the two automorphisms on ${0}$. Geometrically, one can think of this groupoid as being formed by taking the five-element set ${\{-2,-1,0,1,2\}}$, and “folding it in half” around the fixed point ${0}$, giving rise to two “full” quotient points ${[1], [2]}$ and one “half” point ${[0]}$. More generally, given a finite group ${G}$ acting on a finite set ${X}$, and forming the associated multiply connected groupoid, the cardinality up to isomorphism of this groupoid will be ${|X|/|G|}$, since each element ${x}$ of ${X}$ will have ${|G|}$ isomorphisms on it (whether they be to the same element ${x}$, or to other elements of ${X}$).

The definition (2) can also make sense for some infinite groupoids; to my knowledge this was first explicitly done in this paper of Baez and Dolan. Consider for instance the category ${\mathbf{FinSet}}$ of finite sets, with isomorphisms given by bijections as in Example 3. Every finite set is isomorphic to ${\{1,\dots,n\}}$ for some natural number ${n}$, so the equivalence classes of ${\mathbf{FinSet}}$ may be indexed by the natural numbers. The automorphism group ${S_n}$ of ${\{1,\dots,n\}}$ has order ${n!}$, so the cardinality of ${\mathbf{FinSet}}$ up to isomorphism is

$\displaystyle \sum_{n=0}^\infty \frac{1}{n!} = e.$

(This fact is sometimes loosely stated as “the number of finite sets is ${e}$“, but I view this statement as somewhat misleading if the qualifier “up to isomorphism” is not added.) Similarly, when one allows for multiple isomorphisms from a group to itself, the number of groups of order four up to isomorphism is now

$\displaystyle \frac{1}{2} + \frac{1}{6} = \frac{2}{3}$

because the cyclic group ${C_4}$ has two automorphisms, whereas the Klein four-group ${C_2 \times C_2}$ has six.

In the case that the cardinality of a groupoid ${X}$ up to isomorphism is finite and non-empty, one can now define the notion of a random isomorphism class ${[\mathbf{x}]}$ in ${X/\sim}$ drawn “uniformly up to isomorphism”, by requiring the probability of attaining any given isomorphism class ${[x]}$ to be

$\displaystyle {\mathbf P}([\mathbf{x}] = [x]) = \frac{1 / |\mathrm{Aut}(x)|}{\sum_{[y] \in X/\sim} 1/|\mathrm{Aut}(y)|},$

thus the probability of being isomorphic to a given element ${x}$ will be inversely proportional to the number of automorphisms that ${x}$ has. For instance, if we take ${X}$ to be the set ${\{-2,-1,0,1,2\}}$ with the simply connected groupoid, ${[\mathbf{x}]}$ will be drawn uniformly from the three available equivalence classes ${[0], [1], [2]}$, with a ${1/3}$ probability of attaining each; but if instead one uses the multiply connected groupoid coming from the action of ${\{-1,+1\}}$, and draws ${[\mathbf{x}]}$ uniformly up to isomorphism, then ${[1]}$ and ${[2]}$ will now be selected with probability ${2/5}$ each, and ${[0]}$ will be selected with probability ${1/5}$. Thus this distribution has accounted for the bias mentioned previously: if a finite group ${G}$ acts on a finite space ${X}$, and ${\mathbf{x}}$ is drawn uniformly from ${X}$, then ${[\mathbf{x}]}$ now still be drawn uniformly up to isomorphism from ${X/G}$, if we use the multiply connected groupoid coming from the ${G}$ action, rather than the simply connected groupoid coming from just the ${G}$-orbit structure on ${X}$.

Using the groupoid of finite sets, we see that a finite set chosen uniformly up to isomorphism will have a cardinality that is distributed according to the Poisson distribution of parameter ${1}$, that is to say it will be of cardinality ${n}$ with probability ${\frac{e^{-1}}{n!}}$.

One important source of groupoids are the fundamental groupoids ${\pi_1(M)}$ of a manifold ${M}$ (one can also consider more general topological spaces than manifolds, but for simplicity we will restrict this discussion to the manifold case), in which the underlying space is simply ${M}$, and the isomorphisms from ${x}$ to ${y}$ are the equivalence classes of paths from ${x}$ to ${y}$ up to homotopy; in particular, the automorphism group of any given point is just the fundamental group of ${M}$ at that base point. The equivalence class ${[x]}$ of a point in ${M}$ is then the connected component of ${x}$ in ${M}$. The cardinality up to isomorphism of the fundamental groupoid is then

$\displaystyle \sum_{M' \in \pi_0(M)} \frac{1}{|\pi_1(M')|}$

where ${\pi_0(M)}$ is the collection of connected components ${M'}$ of ${M}$, and ${|\pi_1(M')|}$ is the order of the fundamental group of ${M'}$. Thus, simply connected components of ${M}$ count for a full unit of cardinality, whereas multiply connected components (which can be viewed as quotients of their simply connected cover by their fundamental group) will count for a fractional unit of cardinality, inversely to the order of their fundamental group.

This notion of cardinality up to isomorphism of a groupoid behaves well with respect to various basic notions. For instance, suppose one has an ${n}$-fold covering map ${\pi: X \rightarrow Y}$ of one finite groupoid ${Y}$ by another ${X}$. This means that ${\pi}$ is a functor that is surjective, with all preimages of cardinality ${n}$, with the property that given any pair ${y,y'}$ in the base space ${Y}$ and any ${x}$ in the preimage ${\pi^{-1}(\{y\})}$ of ${y}$, every isomorphism ${f \in \mathrm{Iso}(y \rightarrow y')}$ has a unique lift ${\tilde f \in \mathrm{Iso}(x \rightarrow x')}$ from the given initial point ${x}$ (and some ${x'}$ in the preimage of ${y'}$). Then one can check that the cardinality up to isomorphism of ${X}$ is ${n}$ times the cardinality up to isomorphism of ${Y}$, which fits well with the geometric picture of ${X}$ as the ${n}$-fold cover of ${Y}$. (For instance, if one covers a manifold ${M}$ with finite fundamental group by its universal cover, this is a ${|\pi_1(M)|}$-fold cover, the base has cardinality ${1/|\pi_1(M)|}$ up to isomorphism, and the universal cover has cardinality one up to isomorphism.) Related to this, if one draws an equivalence class ${[\mathrm{x}]}$ of ${X}$ uniformly up to isomorphism, then ${\pi([\mathrm{x}])}$ will be an equivalence class of ${Y}$ drawn uniformly up to isomorphism also.

Indeed, one can show that this notion of cardinality up to isomorphism for groupoids is uniquely determined by a small number of axioms such as these (similar to the axioms that determine Euler characteristic); see this blog post of Qiaochu Yuan for details.

The probability distributions on isomorphism classes described by the above recipe seem to arise naturally in many applications. For instance, if one draws a profinite abelian group up to isomorphism at random in this fashion (so that each isomorphism class ${[G]}$ of a profinite abelian group ${G}$ occurs with probability inversely proportional to the number of automorphisms of this group), then the resulting distribution is known as the Cohen-Lenstra distribution, and seems to emerge as the natural asymptotic distribution of many randomly generated profinite abelian groups in number theory and combinatorics, such as the class groups of random quadratic fields; see this previous blog post for more discussion. For a simple combinatorial example, the set of fixed points of a random permutation on ${n}$ elements will have a cardinality that converges in distribution to the Poisson distribution of rate ${1}$ (as discussed in this previous post), thus we see that the fixed points of a large random permutation asymptotically are distributed uniformly up to isomorphism. I’ve been told that this notion of cardinality up to isomorphism is also particularly compatible with stacks (which are a good framework to describe such objects as moduli spaces of algebraic varieties up to isomorphism), though I am not sufficiently acquainted with this theory to say much more than this.

Just a short post to note that Norwegian Academy of Science and Letters has just announced that the 2017 Abel prize has been awarded to Yves Meyer, “for his pivotal role in the development of the mathematical theory of wavelets”.  The actual prize ceremony will be at Oslo in May.

I am actually in Oslo myself currently, having just presented Meyer’s work at the announcement ceremony (and also having written a brief description of some of his work).  The Abel prize has a somewhat unintuitive (and occasionally misunderstood) arrangement in which the presenter of the work of the prize is selected independently of the winner of the prize (I think in part so that the choice of presenter gives no clues as to the identity of the laureate).  In particular, like other presenters before me (which in recent years have included Timothy Gowers, Jordan Ellenberg, and Alex Bellos), I agreed to present the laureate’s work before knowing who the laureate was!  But in this case the task was very easy, because Meyer’s areas of (both pure and applied) harmonic analysis and PDE fell rather squarely within my own area of expertise.  (I had previously written about some other work of Meyer in this blog post.)  Indeed I had learned about Meyer’s wavelet constructions as a graduate student while taking a course from Ingrid Daubechies.   Daubechies also made extremely important contributions to the theory of wavelets, but due to a conflict of interest (as per the guidelines for the prize committee) arising from Daubechies’ presidency of the International Mathematical Union (which nominates the majority of the members of the Abel prize committee, who then serve for two years) from 2011 to 2014 (and her continuing service ex officio on the IMU executive committee from 2015 to 2018), she will not be eligible for the prize until 2021 at the earliest, and so I do not think this prize should be necessarily construed as a judgement on the relative contributions of Meyer and Daubechies to this field.  (In any case I fully agree with the Abel prize committee’s citation of Meyer’s pivotal role in the development of the theory of wavelets.)

[Update, Mar 28: link to prize committee guidelines and clarification of the extent of Daubechies’ conflict of interest added. -T]

Given a function ${f: {\bf N} \rightarrow \{-1,+1\}}$ on the natural numbers taking values in ${+1, -1}$, one can invoke the Furstenberg correspondence principle to locate a measure preserving system ${T \circlearrowright (X, \mu)}$ – a probability space ${(X,\mu)}$ together with a measure-preserving shift ${T: X \rightarrow X}$ (or equivalently, a measure-preserving ${{\bf Z}}$-action on ${(X,\mu)}$) – together with a measurable function (or “observable”) ${F: X \rightarrow \{-1,+1\}}$ that has essentially the same statistics as ${f}$ in the sense that

$\displaystyle \lim \inf_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N f(n+h_1) \dots f(n+h_k)$

$\displaystyle \leq \int_X F(T^{h_1} x) \dots F(T^{h_k} x)\ d\mu(x)$

$\displaystyle \leq \lim \sup_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N f(n+h_1) \dots f(n+h_k)$

for any integers ${h_1,\dots,h_k}$. In particular, one has

$\displaystyle \int_X F(T^{h_1} x) \dots F(T^{h_k} x)\ d\mu(x) = \lim_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N f(n+h_1) \dots f(n+h_k) \ \ \ \ \ (1)$

whenever the limit on the right-hand side exists. We will refer to the system ${T \circlearrowright (X,\mu)}$ together with the designated function ${F}$ as a Furstenberg limit ot the sequence ${f}$. These Furstenberg limits capture some, but not all, of the asymptotic behaviour of ${f}$; roughly speaking, they control the typical “local” behaviour of ${f}$, involving correlations such as ${\frac{1}{N} \sum_{n=1}^N f(n+h_1) \dots f(n+h_k)}$ in the regime where ${h_1,\dots,h_k}$ are much smaller than ${N}$. However, the control on error terms here is usually only qualitative at best, and one usually does not obtain non-trivial control on correlations in which the ${h_1,\dots,h_k}$ are allowed to grow at some significant rate with ${N}$ (e.g. like some power ${N^\theta}$ of ${N}$).

The correspondence principle is discussed in these previous blog posts. One way to establish the principle is by introducing a Banach limit ${p\!-\!\lim: \ell^\infty({\bf N}) \rightarrow {\bf R}}$ that extends the usual limit functional on the subspace of ${\ell^\infty({\bf N})}$ consisting of convergent sequences while still having operator norm one. Such functionals cannot be constructed explicitly, but can be proven to exist (non-constructively and non-uniquely) using the Hahn-Banach theorem; one can also use a non-principal ultrafilter here if desired. One can then seek to construct a system ${T \circlearrowright (X,\mu)}$ and a measurable function ${F: X \rightarrow \{-1,+1\}}$ for which one has the statistics

$\displaystyle \int_X F(T^{h_1} x) \dots F(T^{h_k} x)\ d\mu(x) = p\!-\!\lim_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N f(n+h_1) \dots f(n+h_k) \ \ \ \ \ (2)$

for all ${h_1,\dots,h_k}$. One can explicitly construct such a system as follows. One can take ${X}$ to be the Cantor space ${\{-1,+1\}^{\bf Z}}$ with the product ${\sigma}$-algebra and the shift

$\displaystyle T ( (x_n)_{n \in {\bf Z}} ) := (x_{n+1})_{n \in {\bf Z}}$

with the function ${F: X \rightarrow \{-1,+1\}}$ being the coordinate function at zero:

$\displaystyle F( (x_n)_{n \in {\bf Z}} ) := x_0$

(so in particular ${F( T^h (x_n)_{n \in {\bf Z}} ) = x_h}$ for any ${h \in {\bf Z}}$). The only thing remaining is to construct the invariant measure ${\mu}$. In order to be consistent with (2), one must have

$\displaystyle \mu( \{ (x_n)_{n \in {\bf Z}}: x_{h_j} = \epsilon_j \forall 1 \leq j \leq k \} )$

$\displaystyle = p\!-\!\lim_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N 1_{f(n+h_1)=\epsilon_1} \dots 1_{f(n+h_k)=\epsilon_k}$

for any distinct integers ${h_1,\dots,h_k}$ and signs ${\epsilon_1,\dots,\epsilon_k}$. One can check that this defines a premeasure on the Boolean algebra of ${\{-1,+1\}^{\bf Z}}$ defined by cylinder sets, and the existence of ${\mu}$ then follows from the Hahn-Kolmogorov extension theorem (or the closely related Kolmogorov extension theorem). One can then check that the correspondence (2) holds, and that ${\mu}$ is translation-invariant; the latter comes from the translation invariance of the (Banach-)Césaro averaging operation ${f \mapsto p\!-\!\lim_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N f(n)}$. A variant of this construction shows that the Furstenberg limit is unique up to equivalence if and only if all the limits appearing in (1) actually exist.

One can obtain a slightly tighter correspondence by using a smoother average than the Césaro average. For instance, one can use the logarithmic Césaro averages ${\lim_{N \rightarrow \infty} \frac{1}{\log N}\sum_{n=1}^N \frac{f(n)}{n}}$ in place of the Césaro average ${\sum_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N f(n)}$, thus one replaces (2) by

$\displaystyle \int_X F(T^{h_1} x) \dots F(T^{h_k} x)\ d\mu(x)$

$\displaystyle = p\!-\!\lim_{N \rightarrow \infty} \frac{1}{\log N} \sum_{n=1}^N \frac{f(n+h_1) \dots f(n+h_k)}{n}.$

Whenever the Césaro average of a bounded sequence ${f: {\bf N} \rightarrow {\bf R}}$ exists, then the logarithmic Césaro average exists and is equal to the Césaro average. Thus, a Furstenberg limit constructed using logarithmic Banach-Césaro averaging still obeys (1) for all ${h_1,\dots,h_k}$ when the right-hand side limit exists, but also obeys the more general assertion

$\displaystyle \int_X F(T^{h_1} x) \dots F(T^{h_k} x)\ d\mu(x)$

$\displaystyle = \lim_{N \rightarrow \infty} \frac{1}{\log N} \sum_{n=1}^N \frac{f(n+h_1) \dots f(n+h_k)}{n}$

whenever the limit of the right-hand side exists.

In a recent paper of Frantizinakis, the Furstenberg limits of the Liouville function ${\lambda}$ (with logarithmic averaging) were studied. Some (but not all) of the known facts and conjectures about the Liouville function can be interpreted in the Furstenberg limit. For instance, in a recent breakthrough result of Matomaki and Radziwill (discussed previously here), it was shown that the Liouville function exhibited cancellation on short intervals in the sense that

$\displaystyle \lim_{H \rightarrow \infty} \limsup_{X \rightarrow \infty} \frac{1}{X} \int_X^{2X} \frac{1}{H} |\sum_{x \leq n \leq x+H} \lambda(n)|\ dx = 0.$

In terms of Furstenberg limits of the Liouville function, this assertion is equivalent to the assertion that

$\displaystyle \lim_{H \rightarrow \infty} \int_X |\frac{1}{H} \sum_{h=1}^H F(T^h x)|\ d\mu(x) = 0$

for all Furstenberg limits ${T \circlearrowright (X,\mu), F}$ of Liouville (including those without logarithmic averaging). Invoking the mean ergodic theorem (discussed in this previous post), this assertion is in turn equivalent to the observable ${F}$ that corresponds to the Liouville function being orthogonal to the invariant factor ${L^\infty(X,\mu)^{\bf Z} = \{ g \in L^\infty(X,\mu): g \circ T = g \}}$ of ${X}$; equivalently, the first Gowers-Host-Kra seminorm ${\|F\|_{U^1(X)}}$ of ${F}$ (as defined for instance in this previous post) vanishes. The Chowla conjecture, which asserts that

$\displaystyle \lim_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N \lambda(n+h_1) \dots \lambda(n+h_k) = 0$

for all distinct integers ${h_1,\dots,h_k}$, is equivalent to the assertion that all the Furstenberg limits of Liouville are equivalent to the Bernoulli system (${\{-1,+1\}^{\bf Z}}$ with the product measure arising from the uniform distribution on ${\{-1,+1\}}$, with the shift ${T}$ and observable ${F}$ as before). Similarly, the logarithmically averaged Chowla conjecture

$\displaystyle \lim_{N \rightarrow \infty} \frac{1}{\log N} \sum_{n=1}^N \frac{\lambda(n+h_1) \dots \lambda(n+h_k)}{n} = 0$

is equivalent to the assertion that all the Furstenberg limits of Liouville with logarithmic averaging are equivalent to the Bernoulli system. Recently, I was able to prove the two-point version

$\displaystyle \lim_{N \rightarrow \infty} \frac{1}{\log N} \sum_{n=1}^N \frac{\lambda(n) \lambda(n+h)}{n} = 0 \ \ \ \ \ (3)$

of the logarithmically averaged Chowla conjecture, for any non-zero integer ${h}$; this is equivalent to the perfect strong mixing property

$\displaystyle \int_X F(x) F(T^h x)\ d\mu(x) = 0$

for any Furstenberg limit of Liouville with logarithmic averaging, and any ${h \neq 0}$.

The situation is more delicate with regards to the Sarnak conjecture, which is equivalent to the assertion that

$\displaystyle \lim_{N \rightarrow \infty} \frac{1}{N} \sum_{n=1}^N \lambda(n) f(n) = 0$

for any zero-entropy sequence ${f: {\bf N} \rightarrow {\bf R}}$ (see this previous blog post for more discussion). Morally speaking, this conjecture should be equivalent to the assertion that any Furstenberg limit of Liouville is disjoint from any zero entropy system, but I was not able to formally establish an implication in either direction due to some technical issues regarding the fact that the Furstenberg limit does not directly control long-range correlations, only short-range ones. (There are however ergodic theoretic interpretations of the Sarnak conjecture that involve the notion of generic points; see this paper of El Abdalaoui, Lemancyk, and de la Rue.) But the situation is currently better with the logarithmically averaged Sarnak conjecture

$\displaystyle \lim_{N \rightarrow \infty} \frac{1}{\log N} \sum_{n=1}^N \frac{\lambda(n) f(n)}{n} = 0,$

as I was able to show that this conjecture was equivalent to the logarithmically averaged Chowla conjecture, and hence to all Furstenberg limits of Liouville with logarithmic averaging being Bernoulli; I also showed the conjecture was equivalent to local Gowers uniformity of the Liouville function, which is in turn equivalent to the function ${F}$ having all Gowers-Host-Kra seminorms vanishing in every Furstenberg limit with logarithmic averaging. In this recent paper of Frantzikinakis, this analysis was taken further, showing that the logarithmically averaged Chowla and Sarnak conjectures were in fact equivalent to the much milder seeming assertion that all Furstenberg limits with logarithmic averaging were ergodic.

Actually, the logarithmically averaged Furstenberg limits have more structure than just a ${{\bf Z}}$-action on a measure preserving system ${(X,\mu)}$ with a single observable ${F}$. Let ${Aff_+({\bf Z})}$ denote the semigroup of affine maps ${n \mapsto an+b}$ on the integers with ${a,b \in {\bf Z}}$ and ${a}$ positive. Also, let ${\hat {\bf Z}}$ denote the profinite integers (the inverse limit of the cyclic groups ${{\bf Z}/q{\bf Z}}$). Observe that ${Aff_+({\bf Z})}$ acts on ${\hat {\bf Z}}$ by taking the inverse limit of the obvious actions of ${Aff_+({\bf Z})}$ on ${{\bf Z}/q{\bf Z}}$.

Proposition 1 (Enriched logarithmically averaged Furstenberg limit of Liouville) Let ${p\!-\!\lim}$ be a Banach limit. Then there exists a probability space ${(X,\mu)}$ with an action ${\phi \mapsto T^\phi}$ of the affine semigroup ${Aff_+({\bf Z})}$, as well as measurable functions ${F: X \rightarrow \{-1,+1\}}$ and ${M: X \rightarrow \hat {\bf Z}}$, with the following properties:

• (i) (Affine Furstenberg limit) For any ${\phi_1,\dots,\phi_k \in Aff_+({\bf Z})}$, and any congruence class ${a\ (q)}$, one has

$\displaystyle p\!-\!\lim_{N \rightarrow \infty} \frac{1}{\log N} \sum_{n=1}^N \frac{\lambda(\phi_1(n)) \dots \lambda(\phi_k(n)) 1_{n = a\ (q)}}{n}$

$\displaystyle = \int_X F( T^{\phi_1}(x) ) \dots F( T^{\phi_k}(x) ) 1_{M(x) = a\ (q)}\ d\mu(x).$

• (ii) (Equivariance of ${M}$) For any ${\phi \in Aff_+({\bf Z})}$, one has

$\displaystyle M( T^\phi(x) ) = \phi( M(x) )$

for ${\mu}$-almost every ${x \in X}$.

• (iii) (Multiplicativity at fixed primes) For any prime ${p}$, one has

$\displaystyle F( T^{p\cdot} x ) = - F(x)$

for ${\mu}$-almost every ${x \in X}$, where ${p \cdot \in Aff_+({\bf Z})}$ is the dilation map ${n \mapsto pn}$.

• (iv) (Measure pushforward) If ${\phi \in Aff_+({\bf Z})}$ is of the form ${\phi(n) = an+b}$ and ${S_\phi \subset X}$ is the set ${S_\phi = \{ x \in X: M(x) \in \phi(\hat {\bf Z}) \}}$, then the pushforward ${T^\phi_* \mu}$ of ${\mu}$ by ${\phi}$ is equal to ${a \mu\downharpoonright_{S_\phi}}$, that is to say one has

$\displaystyle \mu( (T^\phi)^{-1}(E) ) = a \mu( E \cap S_\phi )$

for every measurable ${E \subset X}$.

Note that ${{\bf Z}}$ can be viewed as the subgroup of ${Aff_+({\bf Z})}$ consisting of the translations ${n \mapsto n + b}$. If one only keeps the ${{\bf Z}}$-portion of the ${Aff_+({\bf Z})}$ action and forgets the rest (as well as the function ${M}$) then the action becomes measure-preserving, and we recover an ordinary Furstenberg limit with logarithmic averaging. However, the additional structure here can be quite useful; for instance, one can transfer the proof of (3) to this setting, which we sketch below the fold, after proving the proposition.

The observable ${M}$, roughly speaking, means that points ${x}$ in the Furstenberg limit ${X}$ constructed by this proposition are still “virtual integers” in the sense that one can meaningfully compute the residue class of ${x}$ modulo any natural number modulus ${q}$, by first applying ${M}$ and then reducing mod ${q}$. The action of ${Aff_+({\bf Z})}$ means that one can also meaningfully multiply ${x}$ by any natural number, and translate it by any integer. As with other applications of the correspondence principle, the main advantage of moving to this more “virtual” setting is that one now acquires a probability measure ${\mu}$, so that the tools of ergodic theory can be readily applied.

Given a random variable ${X}$ that takes on only finitely many values, we can define its Shannon entropy by the formula

$\displaystyle H(X) := \sum_x \mathbf{P}(X=x) \log \frac{1}{\mathbf{P}(X=x)}$

with the convention that ${0 \log \frac{1}{0} = 0}$. (In some texts, one uses the logarithm to base ${2}$ rather than the natural logarithm, but the choice of base will not be relevant for this discussion.) This is clearly a nonnegative quantity. Given two random variables ${X,Y}$ taking on finitely many values, the joint variable ${(X,Y)}$ is also a random variable taking on finitely many values, and also has an entropy ${H(X,Y)}$. It obeys the Shannon inequalities

$\displaystyle H(X), H(Y) \leq H(X,Y) \leq H(X) + H(Y)$

so we can define some further nonnegative quantities, the mutual information

$\displaystyle I(X:Y) := H(X) + H(Y) - H(X,Y)$

and the conditional entropies

$\displaystyle H(X|Y) := H(X,Y) - H(Y); \quad H(Y|X) := H(X,Y) - H(X).$

More generally, given three random variables ${X,Y,Z}$, one can define the conditional mutual information

$\displaystyle I(X:Y|Z) := H(X|Z) + H(Y|Z) - H(X,Y|Z)$

and the final of the Shannon entropy inequalities asserts that this quantity is also non-negative.

The mutual information ${I(X:Y)}$ is a measure of the extent to which ${X}$ and ${Y}$ fail to be independent; indeed, it is not difficult to show that ${I(X:Y)}$ vanishes if and only if ${X}$ and ${Y}$ are independent. Similarly, ${I(X:Y|Z)}$ vanishes if and only if ${X}$ and ${Y}$ are conditionally independent relative to ${Z}$. At the other extreme, ${H(X|Y)}$ is a measure of the extent to which ${X}$ fails to depend on ${Y}$; indeed, it is not difficult to show that ${H(X|Y)=0}$ if and only if ${X}$ is determined by ${Y}$ in the sense that there is a deterministic function ${f}$ such that ${X = f(Y)}$. In a related vein, if ${X}$ and ${X'}$ are equivalent in the sense that there are deterministic functional relationships ${X = f(X')}$, ${X' = g(X)}$ between the two variables, then ${X}$ is interchangeable with ${X'}$ for the purposes of computing the above quantities, thus for instance ${H(X) = H(X')}$, ${H(X,Y) = H(X',Y)}$, ${I(X:Y) = I(X':Y)}$, ${I(X:Y|Z) = I(X':Y|Z)}$, etc..

One can get some initial intuition for these information-theoretic quantities by specialising to a simple situation in which all the random variables ${X}$ being considered come from restricting a single random (and uniformly distributed) boolean function ${F: \Omega \rightarrow \{0,1\}}$ on a given finite domain ${\Omega}$ to some subset ${A}$ of ${\Omega}$:

$\displaystyle X = F \downharpoonright_A.$

In this case, ${X}$ has the law of a random uniformly distributed boolean function from ${A}$ to ${\{0,1\}}$, and the entropy here can be easily computed to be ${|A| \log 2}$, where ${|A|}$ denotes the cardinality of ${A}$. If ${X}$ is the restriction of ${F}$ to ${A}$, and ${Y}$ is the restriction of ${F}$ to ${B}$, then the joint variable ${(X,Y)}$ is equivalent to the restriction of ${F}$ to ${A \cup B}$. If one discards the normalisation factor ${\log 2}$, one then obtains the following dictionary between entropy and the combinatorics of finite sets:

 Random variables ${X,Y,Z}$ Finite sets ${A,B,C}$ Entropy ${H(X)}$ Cardinality ${|A|}$ Joint variable ${(X,Y)}$ Union ${A \cup B}$ Mutual information ${I(X:Y)}$ Intersection cardinality ${|A \cap B|}$ Conditional entropy ${H(X|Y)}$ Set difference cardinality ${|A \backslash B|}$ Conditional mutual information ${I(X:Y|Z)}$ ${|(A \cap B) \backslash C|}$ ${X, Y}$ independent ${A, B}$ disjoint ${X}$ determined by ${Y}$ ${A}$ a subset of ${B}$ ${X,Y}$ conditionally independent relative to ${Z}$ ${A \cap B \subset C}$

Every (linear) inequality or identity about entropy (and related quantities, such as mutual information) then specialises to a combinatorial inequality or identity about finite sets that is easily verified. For instance, the Shannon inequality ${H(X,Y) \leq H(X)+H(Y)}$ becomes the union bound ${|A \cup B| \leq |A| + |B|}$, and the definition of mutual information becomes the inclusion-exclusion formula

$\displaystyle |A \cap B| = |A| + |B| - |A \cup B|.$

For a more advanced example, consider the data processing inequality that asserts that if ${X, Z}$ are conditionally independent relative to ${Y}$, then ${I(X:Z) \leq I(X:Y)}$. Specialising to sets, this now says that if ${A, C}$ are disjoint outside of ${B}$, then ${|A \cap C| \leq |A \cap B|}$; this can be made apparent by considering the corresponding Venn diagram. This dictionary also suggests how to prove the data processing inequality using the existing Shannon inequalities. Firstly, if ${A}$ and ${C}$ are not necessarily disjoint outside of ${B}$, then a consideration of Venn diagrams gives the more general inequality

$\displaystyle |A \cap C| \leq |A \cap B| + |(A \cap C) \backslash B|$

and a further inspection of the diagram then reveals the more precise identity

$\displaystyle |A \cap C| + |(A \cap B) \backslash C| = |A \cap B| + |(A \cap C) \backslash B|.$

Using the dictionary in the reverse direction, one is then led to conjecture the identity

$\displaystyle I( X : Z ) + I( X : Y | Z ) = I( X : Y ) + I( X : Z | Y )$

which (together with non-negativity of conditional mutual information) implies the data processing inequality, and this identity is in turn easily established from the definition of mutual information.

On the other hand, not every assertion about cardinalities of sets generalises to entropies of random variables that are not arising from restricting random boolean functions to sets. For instance, a basic property of sets is that disjointness from a given set ${C}$ is preserved by unions:

$\displaystyle A \cap C = B \cap C = \emptyset \implies (A \cup B) \cap C = \emptyset.$

Indeed, one has the union bound

$\displaystyle |(A \cup B) \cap C| \leq |A \cap C| + |B \cap C|. \ \ \ \ \ (1)$

Applying the dictionary in the reverse direction, one might now conjecture that if ${X}$ was independent of ${Z}$ and ${Y}$ was independent of ${Z}$, then ${(X,Y)}$ should also be independent of ${Z}$, and furthermore that

$\displaystyle I(X,Y:Z) \leq I(X:Z) + I(Y:Z)$

but these statements are well known to be false (for reasons related to pairwise independence of random variables being strictly weaker than joint independence). For a concrete counterexample, one can take ${X, Y \in {\bf F}_2}$ to be independent, uniformly distributed random elements of the finite field ${{\bf F}_2}$ of two elements, and take ${Z := X+Y}$ to be the sum of these two field elements. One can easily check that each of ${X}$ and ${Y}$ is separately independent of ${Z}$, but the joint variable ${(X,Y)}$ determines ${Z}$ and thus is not independent of ${Z}$.

From the inclusion-exclusion identities

$\displaystyle |A \cap C| = |A| + |C| - |A \cup C|$

$\displaystyle |B \cap C| = |B| + |C| - |B \cup C|$

$\displaystyle |(A \cup B) \cap C| = |A \cup B| + |C| - |A \cup B \cup C|$

$\displaystyle |A \cap B \cap C| = |A| + |B| + |C| - |A \cup B| - |B \cup C| - |A \cup C|$

$\displaystyle + |A \cup B \cup C|$

one can check that (1) is equivalent to the trivial lower bound ${|A \cap B \cap C| \geq 0}$. The basic issue here is that in the dictionary between entropy and combinatorics, there is no satisfactory entropy analogue of the notion of a triple intersection ${A \cap B \cap C}$. (Even the double intersection ${A \cap B}$ only exists information theoretically in a “virtual” sense; the mutual information ${I(X:Y)}$ allows one to “compute the entropy” of this “intersection”, but does not actually describe this intersection itself as a random variable.)

However, this issue only arises with three or more variables; it is not too difficult to show that the only linear equalities and inequalities that are necessarily obeyed by the information-theoretic quantities ${H(X), H(Y), H(X,Y), I(X:Y), H(X|Y), H(Y|X)}$ associated to just two variables ${X,Y}$ are those that are also necessarily obeyed by their combinatorial analogues ${|A|, |B|, |A \cup B|, |A \cap B|, |A \backslash B|, |B \backslash A|}$. (See for instance the Venn diagram at the Wikipedia page for mutual information for a pictorial summation of this statement.)

One can work with a larger class of special cases of Shannon entropy by working with random linear functions rather than random boolean functions. Namely, let ${S}$ be some finite-dimensional vector space over a finite field ${{\mathbf F}}$, and let ${f: S \rightarrow {\mathbf F}}$ be a random linear functional on ${S}$, selected uniformly among all such functions. Every subspace ${U}$ of ${S}$ then gives rise to a random variable ${X = X_U: U \rightarrow {\mathbf F}}$ formed by restricting ${f}$ to ${U}$. This random variable is also distributed uniformly amongst all linear functions on ${U}$, and its entropy can be easily computed to be ${\mathrm{dim}(U) \log |\mathbf{F}|}$. Given two random variables ${X, Y}$ formed by restricting ${f}$ to ${U, V}$ respectively, the joint random variable ${(X,Y)}$ determines the random linear function ${f}$ on the union ${U \cup V}$ on the two spaces, and thus by linearity on the Minkowski sum ${U+V}$ as well; thus ${(X,Y)}$ is equivalent to the restriction of ${f}$ to ${U+V}$. In particular, ${H(X,Y) = \mathrm{dim}(U+V) \log |\mathbf{F}|}$. This implies that ${I(X:Y) = \mathrm{dim}(U \cap V) \log |\mathbf{F}|}$ and also ${H(X|Y) = \mathrm{dim}(\pi_V(U)) \log |\mathbf{F}|}$, where ${\pi_V: S \rightarrow S/V}$ is the quotient map. After discarding the normalising constant ${\log |\mathbf{F}|}$, this leads to the following dictionary between information theoretic quantities and linear algebra quantities, analogous to the previous dictionary:

 Random variables ${X,Y,Z}$ Subspaces ${U,V,W}$ Entropy ${H(X)}$ Dimension ${\mathrm{dim}(U)}$ Joint variable ${(X,Y)}$ Sum ${U+V}$ Mutual information ${I(X:Y)}$ Dimension of intersection ${\mathrm{dim}(U \cap V)}$ Conditional entropy ${H(X|Y)}$ Dimension of projection ${\mathrm{dim}(\pi_V(U))}$ Conditional mutual information ${I(X:Y|Z)}$ ${\mathrm{dim}(\pi_W(U) \cap \pi_W(V))}$ ${X, Y}$ independent ${U, V}$ transverse (${U \cap V = \{0\}}$) ${X}$ determined by ${Y}$ ${U}$ a subspace of ${V}$ ${X,Y}$ conditionally independent relative to ${Z}$ ${\pi_W(U)}$, ${\pi_W(V)}$ transverse.

The combinatorial dictionary can be regarded as a specialisation of the linear algebra dictionary, by taking ${S}$ to be the vector space ${\mathbf{F}_2^\Omega}$ over the finite field ${\mathbf{F}_2}$ of two elements, and only considering those subspaces ${U}$ that are coordinate subspaces ${U = {\bf F}_2^A}$ associated to various subsets ${A}$ of ${\Omega}$.

As before, every linear inequality or equality that is valid for the information-theoretic quantities discussed above, is automatically valid for the linear algebra counterparts for subspaces of a vector space over a finite field by applying the above specialisation (and dividing out by the normalising factor of ${\log |\mathbf{F}|}$). In fact, the requirement that the field be finite can be removed by applying the compactness theorem from logic (or one of its relatives, such as Los’s theorem on ultraproducts, as done in this previous blog post).

The linear algebra model captures more of the features of Shannon entropy than the combinatorial model. For instance, in contrast to the combinatorial case, it is possible in the linear algebra setting to have subspaces ${U,V,W}$ such that ${U}$ and ${V}$ are separately transverse to ${W}$, but their sum ${U+V}$ is not; for instance, in a two-dimensional vector space ${{\bf F}^2}$, one can take ${U,V,W}$ to be the one-dimensional subspaces spanned by ${(0,1)}$, ${(1,0)}$, and ${(1,1)}$ respectively. Note that this is essentially the same counterexample from before (which took ${{\bf F}}$ to be the field of two elements). Indeed, one can show that any necessarily true linear inequality or equality involving the dimensions of three subspaces ${U,V,W}$ (as well as the various other quantities on the above table) will also be necessarily true when applied to the entropies of three discrete random variables ${X,Y,Z}$ (as well as the corresponding quantities on the above table).

However, the linear algebra model does not completely capture the subtleties of Shannon entropy once one works with four or more variables (or subspaces). This was first observed by Ingleton, who established the dimensional inequality

$\displaystyle \mathrm{dim}(U \cap V) \leq \mathrm{dim}(\pi_W(U) \cap \pi_W(V)) + \mathrm{dim}(\pi_X(U) \cap \pi_X(V)) + \mathrm{dim}(W \cap X) \ \ \ \ \ (2)$

for any subspaces ${U,V,W,X}$. This is easiest to see when the three terms on the right-hand side vanish; then ${\pi_W(U), \pi_W(V)}$ are transverse, which implies that ${U\cap V \subset W}$; similarly ${U \cap V \subset X}$. But ${W}$ and ${X}$ are transverse, and this clearly implies that ${U}$ and ${V}$ are themselves transverse. To prove the general case of Ingleton’s inequality, one can define ${Y := U \cap V}$ and use ${\mathrm{dim}(\pi_W(Y)) \leq \mathrm{dim}(\pi_W(U) \cap \pi_W(V))}$ (and similarly for ${X}$ instead of ${W}$) to reduce to establishing the inequality

$\displaystyle \mathrm{dim}(Y) \leq \mathrm{dim}(\pi_W(Y)) + \mathrm{dim}(\pi_X(Y)) + \mathrm{dim}(W \cap X) \ \ \ \ \ (3)$

which can be rearranged using ${\mathrm{dim}(\pi_W(Y)) = \mathrm{dim}(Y) - \mathrm{dim}(W) + \mathrm{dim}(\pi_Y(W))}$ (and similarly for ${X}$ instead of ${W}$) and ${\mathrm{dim}(W \cap X) = \mathrm{dim}(W) + \mathrm{dim}(X) - \mathrm{dim}(W + X)}$ as

$\displaystyle \mathrm{dim}(W + X ) \leq \mathrm{dim}(\pi_Y(W)) + \mathrm{dim}(\pi_Y(X)) + \mathrm{dim}(Y)$

but this is clear since ${\mathrm{dim}(W + X ) \leq \mathrm{dim}(\pi_Y(W) + \pi_Y(X)) + \mathrm{dim}(Y)}$.

Returning to the entropy setting, the analogue

$\displaystyle H( V ) \leq H( V | Z ) + H(V | W ) + I(Z:W)$

of (3) is true (exercise!), but the analogue

$\displaystyle I(X:Y) \leq I(X:Y|Z) + I(X:Y|W) + I(Z:W) \ \ \ \ \ (4)$

of Ingleton’s inequality is false in general. Again, this is easiest to see when all the terms on the right-hand side vanish; then ${X,Y}$ are conditionally independent relative to ${Z}$, and relative to ${W}$, and ${Z}$ and ${W}$ are independent, and the claim (4) would then be asserting that ${X}$ and ${Y}$ are independent. While there is no linear counterexample to this statement, there are simple non-linear ones: for instance, one can take ${Z,W}$ to be independent uniform variables from ${\mathbf{F}_2}$, and take ${X}$ and ${Y}$ to be (say) ${ZW}$ and ${(1-Z)(1-W)}$ respectively (thus ${X, Y}$ are the indicators of the events ${Z=W=1}$ and ${Z=W=0}$ respectively). Once one conditions on either ${Z}$ or ${W}$, one of ${X,Y}$ has positive conditional entropy and the other has zero entropy, and so ${X, Y}$ are conditionally independent relative to either ${Z}$ or ${W}$; also, ${Z}$ or ${W}$ are independent of each other. But ${X}$ and ${Y}$ are not independent of each other (they cannot be simultaneously equal to ${1}$). Somehow, the feature of the linear algebra model that is not present in general is that in the linear algebra setting, every pair of subspaces ${U, V}$ has a well-defined intersection ${U \cap V}$ that is also a subspace, whereas for arbitrary random variables ${X, Y}$, there does not necessarily exist the analogue of an intersection, namely a “common information” random variable ${V}$ that has the entropy of ${I(X:Y)}$ and is determined either by ${X}$ or by ${Y}$.

I do not know if there is any simpler model of Shannon entropy that captures all the inequalities available for four variables. One significant complication is that there exist some information inequalities in this setting that are not of Shannon type, such as the Zhang-Yeung inequality

$\displaystyle I(X:Y) \leq 2 I(X:Y|Z) + I(X:Z|Y) + I(Y:Z|X)$

$\displaystyle + I(X:Y|W) + I(Z:W).$

One can however still use these simpler models of Shannon entropy to be able to guess arguments that would work for general random variables. An example of this comes from my paper on the logarithmically averaged Chowla conjecture, in which I showed among other things that

$\displaystyle |\sum_{n \leq x} \frac{\lambda(n) \lambda(n+1)}{n}| \leq \varepsilon x \ \ \ \ \ (5)$

whenever ${x}$ was sufficiently large depending on ${\varepsilon>0}$, where ${\lambda}$ is the Liouville function. The information-theoretic part of the proof was as follows. Given some intermediate scale ${H}$ between ${1}$ and ${x}$, one can form certain random variables ${X_H, Y_H}$. The random variable ${X_H}$ is a sign pattern of the form ${(\lambda(n+1),\dots,\lambda(n+H))}$ where ${n}$ is a random number chosen from ${1}$ to ${x}$ (with logarithmic weighting). The random variable ${Y_H}$ was tuple ${(n \hbox{ mod } p)_{p \sim \varepsilon^2 H}}$ of reductions of ${n}$ to primes ${p}$ comparable to ${\varepsilon^2 H}$. Roughly speaking, what was implicitly shown in the paper (after using the multiplicativity of ${\lambda}$, the circle method, and the Matomaki-Radziwill theorem on short averages of multiplicative functions) is that if the inequality (5) fails, then there was a lower bound

$\displaystyle I( X_H : Y_H ) \gg \varepsilon^7 \frac{H}{\log H}$

on the mutual information between ${X_H}$ and ${Y_H}$. From translation invariance, this also gives the more general lower bound

$\displaystyle I( X_{H_0,H} : Y_H ) \gg \varepsilon^7 \frac{H}{\log H} \ \ \ \ \ (6)$

for any ${H_0}$, where ${X_{H_0,H}}$ denotes the shifted sign pattern ${(\lambda(n+H_0+1),\dots,\lambda(n+H_0+H))}$. On the other hand, one had the entropy bounds

$\displaystyle H( X_{H_0,H} ), H(Y_H) \ll H$

and from concatenating sign patterns one could see that ${X_{H_0,H+H'}}$ is equivalent to the joint random variable ${(X_{H_0,H}, X_{H_0+H,H'})}$ for any ${H_0,H,H'}$. Applying these facts and using an “entropy decrement” argument, I was able to obtain a contradiction once ${H}$ was allowed to become sufficiently large compared to ${\varepsilon}$, but the bound was quite weak (coming ultimately from the unboundedness of ${\sum_{\log H_- \leq j \leq \log H_+} \frac{1}{j \log j}}$ as the interval ${[H_-,H_+]}$ of values of ${H}$ under consideration becomes large), something of the order of ${H \sim \exp\exp\exp(\varepsilon^{-7})}$; the quantity ${H}$ needs at various junctures to be less than a small power of ${\log x}$, so the relationship between ${x}$ and ${\varepsilon}$ becomes essentially quadruple exponential in nature, ${x \sim \exp\exp\exp\exp(\varepsilon^{-7})}$. The basic strategy was to observe that the lower bound (6) causes some slowdown in the growth rate ${H(X_{kH})/kH}$ of the mean entropy, in that this quantity decreased by ${\gg \frac{\varepsilon^7}{\log H}}$ as ${k}$ increased from ${1}$ to ${\log H}$, basically by dividing ${X_{kH}}$ into ${k}$ components ${X_{jH, H}}$, ${j=0,\dots,k-1}$ and observing from (6) each of these shares a bit of common information with the same variable ${Y_H}$. This is relatively clear when one works in a set model, in which ${Y_H}$ is modeled by a set ${B_H}$ of size ${O(H)}$, and ${X_{H_0,H}}$ is modeled by a set of the form

$\displaystyle X_{H_0,H} = \bigcup_{H_0 < h \leq H_0+H} A_h$

for various sets ${A_h}$ of size ${O(1)}$ (also there is some translation symmetry that maps ${A_h}$ to a shift ${A_{h+1}}$ while preserving all of the ${B_H}$).

However, on considering the set model recently, I realised that one can be a little more efficient by exploiting the fact (basically the Chinese remainder theorem) that the random variables ${Y_H}$ are basically jointly independent as ${H}$ ranges over dyadic values that are much smaller than ${\log x}$, which in the set model corresponds to the ${B_H}$ all being disjoint. One can then establish a variant

$\displaystyle I( X_{H_0,H} : Y_H | (Y_{H'})_{H' < H}) \gg \varepsilon^7 \frac{H}{\log H} \ \ \ \ \ (7)$

of (6), which in the set model roughly speaking asserts that each ${B_H}$ claims a portion of the ${\bigcup_{H_0 < h \leq H_0+H} A_h}$ of cardinality ${\gg \varepsilon^7 \frac{H}{\log H}}$ that is not claimed by previous choices of ${B_H}$. This leads to a more efficient contradiction (relying on the unboundedness of ${\sum_{\log H_- \leq j \leq \log H_+} \frac{1}{j}}$ rather than ${\sum_{\log H_- \leq j \leq \log H_+} \frac{1}{j \log j}}$) that looks like it removes one order of exponential growth, thus the relationship between ${x}$ and ${\varepsilon}$ is now ${x \sim \exp\exp\exp(\varepsilon^{-7})}$. Returning to the entropy model, one can use (7) and Shannon inequalities to establish an inequality of the form

$\displaystyle \frac{1}{2H} H(X_{2H} | (Y_{H'})_{H' \leq 2H}) \leq \frac{1}{H} H(X_{H} | (Y_{H'})_{H' \leq H}) - \frac{c \varepsilon^7}{\log H}$

for a small constant ${c>0}$, which on iterating and using the boundedness of ${\frac{1}{H} H(X_{H} | (Y_{H'})_{H' \leq H})}$ gives the claim. (A modification of this analysis, at least on the level of the back of the envelope calculation, suggests that the Matomaki-Radziwill theorem is needed only for ranges ${H}$ greater than ${\exp( (\log\log x)^{\varepsilon^{7}} )}$ or so, although at this range the theorem is not significantly simpler than the general case).

Daniel Kane and I have just uploaded to the arXiv our paper “A bound on partitioning clusters“, submitted to the Electronic Journal of Combinatorics. In this short and elementary paper, we consider a question that arose from biomathematical applications: given a finite family ${X}$ of sets (or “clusters”), how many ways can there be of partitioning a set ${A \in X}$ in this family as the disjoint union ${A = A_1 \uplus A_2}$ of two other sets ${A_1, A_2}$ in this family? That is to say, what is the best upper bound one can place on the quantity

$\displaystyle | \{ (A,A_1,A_2) \in X^3: A = A_1 \uplus A_2 \}|$

in terms of the cardinality ${|X|}$ of ${X}$? A trivial upper bound would be ${|X|^2}$, since this is the number of possible pairs ${(A_1,A_2)}$, and ${A_1,A_2}$ clearly determine ${A}$. In our paper, we establish the improved bound

$\displaystyle | \{ (A,A_1,A_2) \in X^3: A = A_1 \uplus A_2 \}| \leq |X|^{3/p}$

where ${p}$ is the somewhat strange exponent

$\displaystyle p := \log_3 \frac{27}{4} = 1.73814\dots, \ \ \ \ \ (1)$

so that ${3/p = 1.72598\dots}$. Furthermore, this exponent is best possible!

Actually, the latter claim is quite easy to show: one takes ${X}$ to be all the subsets of ${\{1,\dots,n\}}$ of cardinality either ${n/3}$ or ${2n/3}$, for ${n}$ a multiple of ${3}$, and the claim follows readily from Stirling’s formula. So it is perhaps the former claim that is more interesting (since many combinatorial proof techniques, such as those based on inequalities such as the Cauchy-Schwarz inequality, tend to produce exponents that are rational or at least algebraic). We follow the common, though unintuitive, trick of generalising a problem to make it simpler. Firstly, one generalises the bound to the “trilinear” bound

$\displaystyle | \{ (A_1,A_2,A_3) \in X_1 \times X_2 \times X_3: A_3 = A_1 \uplus A_2 \}|$

$\displaystyle \leq |X_1|^{1/p} |X_2|^{1/p} |X_3|^{1/p}$

for arbitrary finite collections ${X_1,X_2,X_3}$ of sets. One can place all the sets in ${X_1,X_2,X_3}$ inside a single finite set such as ${\{1,\dots,n\}}$, and then by replacing every set ${A_3}$ in ${X_3}$ by its complement in ${\{1,\dots,n\}}$, one can phrase the inequality in the equivalent form

$\displaystyle | \{ (A_1,A_2,A_3) \in X_1 \times X_2 \times X_3: \{1,\dots,n\} =A_1 \uplus A_2 \uplus A_3 \}|$

$\displaystyle \leq |X_1|^{1/p} |X_2|^{1/p} |X_3|^{1/p}$

for arbitrary collections ${X_1,X_2,X_3}$ of subsets of ${\{1,\dots,n\}}$. We generalise further by turning sets into functions, replacing the estimate with the slightly stronger convolution estimate

$\displaystyle f_1 * f_2 * f_3 (1,\dots,1) \leq \|f_1\|_{\ell^p(\{0,1\}^n)} \|f_2\|_{\ell^p(\{0,1\}^n)} \|f_3\|_{\ell^p(\{0,1\}^n)}$

for arbitrary functions ${f_1,f_2,f_3}$ on the Hamming cube ${\{0,1\}^n}$, where the convolution is on the integer lattice ${\bf Z}^n$ rather than on the finite field vector space ${\bf F}_2^n$. The advantage of working in this general setting is that it becomes very easy to apply induction on the dimension ${n}$; indeed, to prove this estimate for arbitrary ${n}$ it suffices to do so for ${n=1}$. This reduces matters to establishing the elementary inequality

$\displaystyle (ab(1-c))^{1/p} + (bc(1-a))^{1/p} + (ca(1-b))^{1/p} \leq 1$

for all ${0 \leq a,b,c \leq 1}$, which can be done by a combination of undergraduate multivariable calculus and a little bit of numerical computation. (The left-hand side turns out to have local maxima at ${(1,1,0), (1,0,1), (0,1,1), (2/3,2/3,2/3)}$, with the latter being the cause of the numerology (1).)

The same sort of argument also gives an energy bound

$\displaystyle E(A,A) \leq |A|^{\log_2 6}$

for any subset ${A \subset \{0,1\}^n}$ of the Hamming cube, where

$\displaystyle E(A,A) := |\{(a_1,a_2,a_3,a_4) \in A^4: a_1+a_2 = a_3 + a_4 \}|$

is the additive energy of ${A}$. The example ${A = \{0,1\}^n}$ shows that the exponent ${\log_2 6}$ cannot be improved.