You are currently browsing the category archive for the ‘math.CO’ category.

[This blog post was written jointly by Terry Tao and Will Sawin.]

In the previous blog post, one of us (Terry) implicitly introduced a notion of rank for tensors which is a little different from the usual notion of tensor rank, and which (following BCCGNSU) we will call “slice rank”. This notion of rank could then be used to encode the Croot-Lev-Pach-Ellenberg-Gijswijt argument that uses the polynomial method to control capsets.

Afterwards, several papers have applied the slice rank method to further problems – to control tri-colored sum-free sets in abelian groups (BCCGNSU, KSS) and from there to the triangle removal lemma in vector spaces over finite fields (FL), to control sunflowers (NS), and to bound progression-free sets in ${p}$-groups (P).

In this post we investigate the notion of slice rank more systematically. In particular, we show how to give lower bounds for the slice rank. In many cases, we can show that the upper bounds on slice rank given in the aforementioned papers are sharp to within a subexponential factor. This still leaves open the possibility of getting a better bound for the original combinatorial problem using the slice rank of some other tensor, but for very long arithmetic progressions (at least eight terms), we show that the slice rank method cannot improve over the trivial bound using any tensor.

It will be convenient to work in a “basis independent” formalism, namely working in the category of abstract finite-dimensional vector spaces over a fixed field ${{\bf F}}$. (In the applications to the capset problem one takes ${{\bf F}={\bf F}_3}$ to be the finite field of three elements, but most of the discussion here applies to arbitrary fields.) Given ${k}$ such vector spaces ${V_1,\dots,V_k}$, we can form the tensor product ${\bigotimes_{i=1}^k V_i}$, generated by the tensor products ${v_1 \otimes \dots \otimes v_k}$ with ${v_i \in V_i}$ for ${i=1,\dots,k}$, subject to the constraint that the tensor product operation ${(v_1,\dots,v_k) \mapsto v_1 \otimes \dots \otimes v_k}$ is multilinear. For each ${1 \leq j \leq k}$, we have the smaller tensor products ${\bigotimes_{1 \leq i \leq k: i \neq j} V_i}$, as well as the ${j^{th}}$ tensor product

$\displaystyle \otimes_j: V_j \times \bigotimes_{1 \leq i \leq k: i \neq j} V_i \rightarrow \bigotimes_{i=1}^k V_i$

defined in the obvious fashion. Elements of ${\bigotimes_{i=1}^k V_i}$ of the form ${v_j \otimes_j v_{\hat j}}$ for some ${v_j \in V_j}$ and ${v_{\hat j} \in \bigotimes_{1 \leq i \leq k: i \neq j} V_i}$ will be called rank one functions, and the slice rank (or rank for short) ${\hbox{rank}(v)}$ of an element ${v}$ of ${\bigotimes_{i=1}^k V_i}$ is defined to be the least nonnegative integer ${r}$ such that ${v}$ is a linear combination of ${r}$ rank one functions. If ${V_1,\dots,V_k}$ are finite-dimensional, then the rank is always well defined as a non-negative integer (in fact it cannot exceed ${\min( \hbox{dim}(V_1), \dots, \hbox{dim}(V_k))}$. It is also clearly subadditive:

$\displaystyle \hbox{rank}(v+w) \leq \hbox{rank}(v) + \hbox{rank}(w). \ \ \ \ \ (1)$

For ${k=1}$, ${\hbox{rank}(v)}$ is ${0}$ when ${v}$ is zero, and ${1}$ otherwise. For ${k=2}$, ${\hbox{rank}(v)}$ is the usual rank of the ${2}$-tensor ${v \in V_1 \otimes V_2}$ (which can for instance be identified with a linear map from ${V_1}$ to the dual space ${V_2^*}$). The usual notion of tensor rank for higher order tensors uses complete tensor products ${v_1 \otimes \dots \otimes v_k}$, ${v_i \in V_i}$ as the rank one objects, rather than ${v_j \otimes_j v_{\hat j}}$, giving a rank that is greater than or equal to the slice rank studied here.

From basic linear algebra we have the following equivalences:

Lemma 1 Let ${V_1,\dots,V_k}$ be finite-dimensional vector spaces over a field ${{\bf F}}$, let ${v}$ be an element of ${V_1 \otimes \dots \otimes V_k}$, and let ${r}$ be a non-negative integer. Then the following are equivalent:

• (i) One has ${\hbox{rank}(v) \leq r}$.
• (ii) One has a representation of the form

$\displaystyle v = \sum_{j=1}^k \sum_{s \in S_j} v_{j,s} \otimes_j v_{\hat j,s}$

where ${S_1,\dots,S_k}$ are finite sets of total cardinality ${|S_1|+\dots+|S_k|}$ at most ${r}$, and for each ${1 \leq j \leq k}$ and ${s \in S_j}$, ${v_{j,s} \in V_j}$ and ${v_{\hat j,s} \in \bigotimes_{1 \leq i \leq k: i \neq j} V_i}$.

• (iii) One has

$\displaystyle v \in \sum_{j=1}^k U_j \otimes_j \bigotimes_{1 \leq i \leq k: i \neq j} V_i$

where for each ${j=1,\dots,k}$, ${U_j}$ is a subspace of ${V_j}$ of total dimension ${\hbox{dim}(U_1)+\dots+\hbox{dim}(U_k)}$ at most ${r}$, and we view ${U_j \otimes_j \bigotimes_{1 \leq i \leq k: i \neq j} V_i}$ as a subspace of ${\bigotimes_{i=1}^k V_i}$ in the obvious fashion.

• (iv) (Dual formulation) There exist subspaces ${W_j}$ of the dual space ${V_j^*}$ for ${j=1,\dots,k}$, of total dimension at least ${\hbox{dim}(V_1)+\dots+\hbox{dim}(V_k) - r}$, such that ${v}$ is orthogonal to ${\bigotimes_{j=1}^k W_j}$, in the sense that one has the vanishing

$\displaystyle \langle \bigotimes_{j=1}^k w_j, v \rangle = 0$

for all ${w_j \in W_j}$, where ${\langle, \rangle: \bigotimes_{j=1}^k V_j^* \times \bigotimes_{j=1}^k V_j \rightarrow {\bf F}}$ is the obvious pairing.

Proof: The equivalence of (i) and (ii) is clear from definition. To get from (ii) to (iii) one simply takes ${U_j}$ to be the span of the ${v_{j,s}}$, and conversely to get from (iii) to (ii) one takes the ${v_{j,s}}$ to be a basis of the ${U_j}$ and computes ${v_{\hat j,s}}$ by using a basis for the tensor product ${\bigotimes_{j=1}^k U_j \otimes_j \bigotimes_{1 \leq i \leq k: i \neq j} V_i}$ consisting entirely of functions of the form ${v_{j,s} \otimes_j e}$ for various ${e}$. To pass from (iii) to (iv) one takes ${W_j}$ to be the annihilator ${\{ w_j \in V_j: \langle w_j, v_j \rangle = 0 \forall v_j \in U_j \}}$ of ${U_j}$, and conversely to pass from (iv) to (iii). $\Box$

One corollary of the formulation (iv), is that the set of tensors of slice rank at most ${r}$ is Zariski closed (if the field ${{\mathbf F}}$ is algebraically closed), and so the slice rank itself is a lower semi-continuous function. This is in contrast to the usual tensor rank, which is not necessarily semicontinuous.

Corollary 2 Let ${V_1,\dots, V_k}$ be finite-dimensional vector spaces over an algebraically closed field ${{\bf F}}$. Let ${r}$ be a nonnegative integer. The set of elements of ${V_1 \otimes \dots \otimes V_k}$ of slice rank at most ${r}$ is closed in the Zariski topology.

Proof: In view of Lemma 1(i and iv), this set is the union over tuples of integers ${d_1,\dots,d_k}$ with ${d_1 + \dots + d_k \geq \hbox{dim}(V_1)+\dots+\hbox{dim}(V_k) - r}$ of the projection from ${\hbox{Gr}(d_1, V_1) \times \dots \times \hbox{Gr}(d_k, V_k) \times ( V_1 \otimes \dots \otimes V_k)}$ of the set of tuples ${(W_1,\dots,W_k, v)}$ with ${ v}$ orthogonal to ${W_1 \times \dots \times W_k}$, where ${\hbox{Gr}(d,V)}$ is the Grassmanian parameterizing ${d}$-dimensional subspaces of ${V}$.

One can check directly that the set of tuples ${(W_1,\dots,W_k, v)}$ with ${ v}$ orthogonal to ${W_1 \times \dots \times W_k}$ is Zariski closed in ${\hbox{Gr}(d_1, V_1) \times \dots \times \hbox{Gr}(d_k, V_k) \times V_1 \otimes \dots \otimes V_k}$ using a set of equations of the form ${\langle \bigotimes_{j=1}^k w_j, v \rangle = 0}$ locally on ${\hbox{Gr}(d_1, V_1) \times \dots \times \hbox{Gr}(d_k, V_k) }$. Hence because the Grassmanian is a complete variety, the projection of this set to ${V_1 \otimes \dots \otimes V_k}$ is also Zariski closed. So the finite union over tuples ${d_1,\dots,d_k}$ of these projections is also Zariski closed.

$\Box$

We also have good behaviour with respect to linear transformations:

Lemma 3 Let ${V_1,\dots,V_k, W_1,\dots,W_k}$ be finite-dimensional vector spaces over a field ${{\bf F}}$, let ${v}$ be an element of ${V_1 \otimes \dots \otimes V_k}$, and for each ${1 \leq j \leq k}$, let ${\phi_j: V_j \rightarrow W_j}$ be a linear transformation, with ${\bigotimes_{j=1}^k \phi_j: \bigotimes_{j=1}^k V_k \rightarrow \bigotimes_{j=1}^k W_k}$ the tensor product of these maps. Then

$\displaystyle \hbox{rank}( (\bigotimes_{j=1}^k \phi_j)(v) ) \leq \hbox{rank}(v). \ \ \ \ \ (2)$

Furthermore, if the ${\phi_j}$ are all injective, then one has equality in (2).

Thus, for instance, the rank of a tensor ${v \in \bigotimes_{j=1}^k V_k}$ is intrinsic in the sense that it is unaffected by any enlargements of the spaces ${V_1,\dots,V_k}$.

Proof: The bound (2) is clear from the formulation (ii) of rank in Lemma 1. For equality, apply (2) to the injective ${\phi_j}$, as well as to some arbitrarily chosen left inverses ${\phi_j^{-1}: W_j \rightarrow V_j}$ of the ${\phi_j}$. $\Box$

Computing the rank of a tensor is difficult in general; however, the problem becomes a combinatorial one if one has a suitably sparse representation of that tensor in some basis, where we will measure sparsity by the property of being an antichain.

Proposition 4 Let ${V_1,\dots,V_k}$ be finite-dimensional vector spaces over a field ${{\bf F}}$. For each ${1 \leq j \leq k}$, let ${(v_{j,s})_{s \in S_j}}$ be a linearly independent set in ${V_j}$ indexed by some finite set ${S_j}$. Let ${\Gamma}$ be a subset of ${S_1 \times \dots \times S_k}$.

Let ${v \in \bigotimes_{j=1}^k V_j}$ be a tensor of the form

$\displaystyle v = \sum_{(s_1,\dots,s_k) \in \Gamma} c_{s_1,\dots,s_k} v_{1,s_1} \otimes \dots \otimes v_{k,s_k} \ \ \ \ \ (3)$

where for each ${(s_1,\dots,s_k)}$, ${c_{s_1,\dots,s_k}}$ is a coefficient in ${{\bf F}}$. Then one has

$\displaystyle \hbox{rank}(v) \leq \min_{\Gamma = \Gamma_1 \cup \dots \cup \Gamma_k} |\pi_1(\Gamma_1)| + \dots + |\pi_k(\Gamma_k)| \ \ \ \ \ (4)$

where the minimum ranges over all coverings of ${\Gamma}$ by sets ${\Gamma_1,\dots,\Gamma_k}$, and ${\pi_j: S_1 \times \dots \times S_k \rightarrow S_j}$ for ${j=1,\dots,k}$ are the projection maps.

Now suppose that the coefficients ${c_{s_1,\dots,s_k}}$ are all non-zero, that each of the ${S_j}$ are equipped with a total ordering ${\leq_j}$, and ${\Gamma'}$ is the set of maximal elements of ${\Gamma}$, thus there do not exist distinct ${(s_1,\dots,s_k) \in \Gamma'}$, ${(t_1,\dots,t_k) \in \Gamma}$ such that ${s_j \leq t_j}$ for all ${j=1,\dots,k}$. Then one has

$\displaystyle \hbox{rank}(v) \geq \min_{\Gamma' = \Gamma_1 \cup \dots \cup \Gamma_k} |\pi_1(\Gamma_1)| + \dots + |\pi_k(\Gamma_k)|. \ \ \ \ \ (5)$

In particular, if ${\Gamma}$ is an antichain (i.e. every element is maximal), then equality holds in (4).

Proof: By Lemma 3 (or by enlarging the bases ${v_{j,s_j}}$), we may assume without loss of generality that each of the ${V_j}$ is spanned by the ${v_{j,s_j}}$. By relabeling, we can also assume that each ${S_j}$ is of the form

$\displaystyle S_j = \{1,\dots,|S_j|\}$

with the usual ordering, and by Lemma 3 we may take each ${V_j}$ to be ${{\bf F}^{|S_j|}}$, with ${v_{j,s_j} = e_{s_j}}$ the standard basis.

Let ${r}$ denote the rank of ${v}$. To show (4), it suffices to show the inequality

$\displaystyle r \leq |\pi_1(\Gamma_1)| + \dots + |\pi_k(\Gamma_k)| \ \ \ \ \ (6)$

for any covering of ${\Gamma}$ by ${\Gamma_1,\dots,\Gamma_k}$. By removing repeated elements we may assume that the ${\Gamma_i}$ are disjoint. For each ${1 \leq j \leq k}$, the tensor

$\displaystyle \sum_{(s_1,\dots,s_k) \in \Gamma_j} c_{s_1,\dots,s_k} e_{s_1} \otimes \dots \otimes e_{s_k}$

can (after collecting terms) be written as

$\displaystyle \sum_{s_j \in \pi_j(\Gamma_j)} e_{s_j} \otimes_j v_{\hat j,s_j}$

for some ${v_{\hat j, s_j} \in \bigotimes_{1 \leq i \leq k: i \neq j} {\bf F}^{|S_i|}}$. Summing and using (1), we conclude the inequality (6).

Now assume that the ${c_{s_1,\dots,s_k}}$ are all non-zero and that ${\Gamma'}$ is the set of maximal elements of ${\Gamma}$. To conclude the proposition, it suffices to show that the reverse inequality

$\displaystyle r \geq |\pi_1(\Gamma_1)| + \dots + |\pi_k(\Gamma_k)| \ \ \ \ \ (7)$

holds for some ${\Gamma_1,\dots,\Gamma_k}$ covering ${\Gamma'}$. By Lemma 1(iv), there exist subspaces ${W_j}$ of ${({\bf F}^{|S_j|})^*}$ whose dimension ${d_j := \hbox{dim}(W_j)}$ sums to

$\displaystyle \sum_{j=1}^k d_j = \sum_{j=1}^k |S_j| - r \ \ \ \ \ (8)$

such that ${v}$ is orthogonal to ${\bigotimes_{j=1}^k W_j}$.

Let ${1 \leq j \leq k}$. Using Gaussian elimination, one can find a basis ${w_{j,1},\dots,w_{j,d_j}}$ of ${W_j}$ whose representation in the standard dual basis ${e^*_{1},\dots,e^*_{|S_j|}}$ of ${({\bf F}^{|S_j|})^*}$ is in row-echelon form. That is to say, there exist natural numbers

$\displaystyle 1 \leq s_{j,1} < \dots < s_{j,d_j} \leq |S_j|$

such that for all ${1 \leq t \leq d_j}$, ${w_{j,t}}$ is a linear combination of the dual vectors ${e^*_{s_{j,t}},\dots,e^*_{|S_j|}}$, with the ${e^*_{s_{j,t}}}$ coefficient equal to one.

We now claim that ${\prod_{j=1}^k \{ s_{j,t}: 1 \leq t \leq d_j \}}$ is disjoint from ${\Gamma'}$. Suppose for contradiction that this were not the case, thus there exists ${1 \leq t_j \leq d_j}$ for each ${1 \leq j \leq k}$ such that

$\displaystyle (s_{1,t_1}, \dots, s_{k,t_k}) \in \Gamma'.$

As ${\Gamma'}$ is the set of maximal elements of ${\Gamma}$, this implies that

$\displaystyle (s'_1,\dots,s'_k) \not \in \Gamma$

for any tuple ${(s'_1,\dots,s'_k) \in \prod_{j=1}^k \{ s_{j,t_j}, \dots, |S_j|\}}$ other than ${(s_{1,t_1}, \dots, s_{k,t_k})}$. On the other hand, we know that ${w_{j,t_j}}$ is a linear combination of ${e^*_{s_{j,t_j}},\dots,e^*_{|S_j|}}$, with the ${e^*_{s_{j,t_j}}}$ coefficient one. We conclude that the tensor product ${\bigotimes_{j=1}^k w_{j,t_j}}$ is equal to

$\displaystyle \bigotimes_{j=1}^k e^*_{s_{j,t_j}}$

plus a linear combination of other tensor products ${\bigotimes_{j=1}^k e^*_{s'_j}}$ with ${(s'_1,\dots,s'_k)}$ not in ${\Gamma}$. Taking inner products with (3), we conclude that ${\langle v, \bigotimes_{j=1}^k w_{j,t_j}\rangle = c_{s_{1,t_1},\dots,s_{k,t_k}} \neq 0}$, contradicting the fact that ${v}$ is orthogonal to ${\prod_{j=1}^k W_j}$. Thus we have ${\prod_{j=1}^k \{ s_{j,t}: 1 \leq t \leq d_j \}}$ disjoint from ${\Gamma'}$.

For each ${1 \leq j \leq k}$, let ${\Gamma_j}$ denote the set of tuples ${(s_1,\dots,s_k)}$ in ${\Gamma'}$ with ${s_j}$ not of the form ${\{ s_{j,t}: 1 \leq t \leq d_j \}}$. From the previous discussion we see that the ${\Gamma_j}$ cover ${\Gamma'}$, and we clearly have ${\pi_j(\Gamma_j) \leq |S_j| - d_j}$, and hence from (8) we have (7) as claimed. $\Box$

As an instance of this proposition, we recover the computation of diagonal rank from the previous blog post:

Example 5 Let ${V_1,\dots,V_k}$ be finite-dimensional vector spaces over a field ${{\bf F}}$ for some ${k \geq 2}$. Let ${d}$ be a natural number, and for ${1 \leq j \leq k}$, let ${e_{j,1},\dots,e_{j,d}}$ be a linearly independent set in ${V_j}$. Let ${c_1,\dots,c_d}$ be non-zero coefficients in ${{\bf F}}$. Then

$\displaystyle \sum_{t=1}^d c_t e_{1,t} \otimes \dots \otimes e_{k,t}$

has rank ${d}$. Indeed, one applies the proposition with ${S_1,\dots,S_k}$ all equal to ${\{1,\dots,d\}}$, with ${\Gamma}$ the diagonal in ${S_1 \times \dots \times S_k}$; this is an antichain if we give one of the ${S_i}$ the standard ordering, and another of the ${S_i}$ the opposite ordering (and ordering the remaining ${S_i}$ arbitrarily). In this case, the ${\pi_j}$ are all bijective, and so it is clear that the minimum in (4) is simply ${d}$.

The combinatorial minimisation problem in the above proposition can be solved asymptotically when working with tensor powers, using the notion of the Shannon entropy ${h(X)}$ of a discrete random variable ${X}$.

Proposition 6 Let ${V_1,\dots,V_k}$ be finite-dimensional vector spaces over a field ${{\bf F}}$. For each ${1 \leq j \leq k}$, let ${(v_{j,s})_{s \in S_j}}$ be a linearly independent set in ${V_j}$ indexed by some finite set ${S_j}$. Let ${\Gamma}$ be a non-empty subset of ${S_1 \times \dots \times S_k}$.

Let ${v \in \bigotimes_{j=1}^k V_j}$ be a tensor of the form (3) for some coefficients ${c_{s_1,\dots,s_k}}$. For each natural number ${n}$, let ${v^{\otimes n}}$ be the tensor power of ${n}$ copies of ${v}$, viewed as an element of ${\bigotimes_{j=1}^k V_j^{\otimes n}}$. Then

$\displaystyle \hbox{rank}(v^{\otimes n}) \leq \exp( (H + o(1)) n ) \ \ \ \ \ (9)$

as ${n \rightarrow \infty}$, where ${H}$ is the quantity

$\displaystyle H = \hbox{sup}_{(X_1,\dots,X_k)} \hbox{min}( h(X_1), \dots, h(X_k) ) \ \ \ \ \ (10)$

and ${(X_1,\dots,X_k)}$ range over the random variables taking values in ${\Gamma}$.

Now suppose that the coefficients ${c_{s_1,\dots,s_k}}$ are all non-zero and that each of the ${S_j}$ are equipped with a total ordering ${\leq_j}$. Let ${\Gamma'}$ be the set of maximal elements of ${\Gamma}$ in the product ordering, and let ${H' = \hbox{sup}_{(X_1,\dots,X_k)} \hbox{min}( h(X_1), \dots, h(X_k) ) }$ where ${(X_1,\dots,X_k)}$ range over random variables taking values in ${\Gamma'}$. Then

$\displaystyle \hbox{rank}(v^{\otimes n}) \geq \exp( (H' + o(1)) n ) \ \ \ \ \ (11)$

as ${n \rightarrow \infty}$. In particular, if the maximizer in (10) is supported on the maximal elements of ${\Gamma}$ (which always holds if ${\Gamma}$ is an antichain in the product ordering), then equality holds in (9).

Proof:

It will suffice to show that

$\displaystyle \min_{\Gamma^n = \Gamma_{n,1} \cup \dots \cup \Gamma_{n,k}} |\pi_{n,1}(\Gamma_{n,1})| + \dots + |\pi_{n,k}(\Gamma_{n,k})| = \exp( (H + o(1)) n ) \ \ \ \ \ (12)$

as ${n \rightarrow \infty}$, where ${\pi_{n,j}: \prod_{i=1}^k S_i^n \rightarrow S_j^n}$ is the projection map. Then the same thing will apply to ${\Gamma'}$ and ${H'}$. Then applying Proposition 4, using the lexicographical ordering on ${S_j^n}$ and noting that, if ${\Gamma'}$ are the maximal elements of ${\Gamma}$, then ${\Gamma'^n}$ are the maximal elements of ${\Gamma^n}$, we obtain both (9) and (11).

We first prove the lower bound. By compactness (and the continuity properties of entropy), we can find a random variable ${(X_1,\dots,X_k)}$ taking values in ${\Gamma}$ such that

$\displaystyle H = \hbox{min}( h(X_1), \dots, h(X_k) ). \ \ \ \ \ (13)$

Let ${\varepsilon = o(1)}$ be a small positive quantity that goes to zero sufficiently slowly with ${n}$. Let ${\Sigma = \Sigma_{X_1,\dots,X_k} \subset \Gamma^n}$ denote the set of all tuples ${(a_1, \dots, \vec a_n)}$ in ${\Gamma^n}$ that are within ${\varepsilon}$ of being distributed according to the law of ${(X_1,\dots,X_k)}$, in the sense that for all ${a \in \Gamma}$, one has

$\displaystyle |\frac{|\{ 1 \leq l \leq n: a_l = a \}|}{n} - {\bf P}( (X_1,\dots,X_k) = a )| \leq \varepsilon.$

By the asymptotic equipartition property, the cardinality of ${\Sigma}$ can be computed to be

$\displaystyle |\Sigma| = \exp( (h( X_1,\dots,X_k)+o(1)) n ) \ \ \ \ \ (14)$

if ${\varepsilon}$ goes to zero slowly enough. Similarly one has

$\displaystyle |\pi_{n,j}(\Sigma)| = \exp( (h( X_j)+o(1)) n ),$

and for each ${s_{n,j} \in \pi_{n,j}(\Sigma)}$, one has

$\displaystyle |\{ \sigma \in \Sigma: \pi_{n,j}(\sigma) = s_{n,j} \}| \leq \exp( (h( X_1,\dots,X_k)-h(X_j)+o(1)) n ). \ \ \ \ \ (15)$

Now let ${\Gamma^n = \Gamma_{n,1} \cup \dots \cup \Gamma_{n,k}}$ be an arbitrary covering of ${\Gamma^n}$. By the pigeonhole principle, there exists ${1 \leq j \leq k}$ such that

$\displaystyle |\Gamma_{n,j} \cap \Sigma| \geq \frac{1}{k} |\Sigma|$

and hence by (14), (15)

$\displaystyle |\pi_{n,j}( \Gamma_{n,j} \cap \Sigma)| \geq \frac{1}{k} \exp( (h( X_j)+o(1)) n )$

which by (13) implies that

$\displaystyle |\pi_{n,1}(\Gamma_{n,1})| + \dots + |\pi_{n,k}(\Gamma_{n,k})| \geq \exp( (H + o(1)) n )$

noting that the ${\frac{1}{k}}$ factor can be absorbed into the ${o(1)}$ error). This gives the lower bound in (12).

Now we prove the upper bound. We can cover ${\Gamma^n}$ by ${O(\exp(o(n))}$ sets of the form ${\Sigma_{X_1,\dots,X_k}}$ for various choices of random variables ${(X_1,\dots,X_k)}$ taking values in ${\Gamma}$. For each such random variable ${(X_1,\dots,X_k)}$, we can find ${1 \leq j \leq k}$ such that ${h(X_j) \leq H}$; we then place all of ${\Sigma_{X_1,\dots,X_k}}$ in ${\Gamma_j}$. It is then clear that the ${\Gamma_j}$ cover ${\Gamma}$ and that

$\displaystyle |\Gamma_j| \leq \exp( (H+o(1)) n )$

for all ${j=1,\dots,n}$, giving the required upper bound. $\Box$

It is of interest to compute the quantity ${H}$ in (10). We have the following criterion for when a maximiser occurs:

Proposition 7 Let ${S_1,\dots,S_k}$ be finite sets, and ${\Gamma \subset S_1 \times \dots \times S_k}$ be non-empty. Let ${H}$ be the quantity in (10). Let ${(X_1,\dots,X_k)}$ be a random variable taking values in ${\Gamma}$, and let ${\Gamma^* \subset \Gamma}$ denote the essential range of ${(X_1,\dots,X_k)}$, that is to say the set of tuples ${(t_1,\dots,t_k)\in \Gamma}$ such that ${{\bf P}( X_1=t_1, \dots, X_k = t_k)}$ is non-zero. Then the following are equivalent:

• (i) ${(X_1,\dots,X_k)}$ attains the maximum in (10).
• (ii) There exist weights ${w_1,\dots,w_k \geq 0}$ and a finite quantity ${D \geq 0}$, such that ${w_j=0}$ whenever ${h(X_j) > \min(h(X_1),\dots,h(X_k))}$, and such that

$\displaystyle \sum_{j=1}^k w_j \log \frac{1}{{\bf P}(X_j = t_j)} \leq D \ \ \ \ \ (16)$

for all ${(t_1,\dots,t_k) \in \Gamma}$, with equality if ${(t_1,\dots,t_k) \in \Gamma^*}$. (In particular, ${w_j}$ must vanish if there exists a ${t_j \in \pi_i(\Gamma)}$ with ${{\bf P}(X_j=t_j)=0}$.)

Furthermore, when (i) and (ii) holds, one has

$\displaystyle D = H \sum_{j=1}^k w_j. \ \ \ \ \ (17)$

Proof: We first show that (i) implies (ii). The function ${p \mapsto p \log \frac{1}{p}}$ is concave on ${[0,1]}$. As a consequence, if we define ${C}$ to be the set of tuples ${(h_1,\dots,h_k) \in [0,+\infty)^k}$ such that there exists a random variable ${(Y_1,\dots,Y_k)}$ taking values in ${\Gamma}$ with ${h(Y_j)=h_j}$, then ${C}$ is convex. On the other hand, by (10), ${C}$ is disjoint from the orthant ${(H,+\infty)^k}$. Thus, by the hyperplane separation theorem, we conclude that there exists a half-space

$\displaystyle \{ (h_1,\dots,h_k) \in {\bf R}^k: w_1 h_1 + \dots + w_k h_k \geq c \},$

where ${w_1,\dots,w_k}$ are reals that are not all zero, and ${c}$ is another real, which contains ${(h(X_1),\dots,h(X_k))}$ on its boundary and ${(H,+\infty)^k}$ in its interior, such that ${C}$ avoids the interior of the half-space. Since ${(h(X_1),\dots,h(X_k))}$ is also on the boundary of ${(H,+\infty)^k}$, we see that the ${w_j}$ are non-negative, and that ${w_j = 0}$ whenever ${h(X_j) \neq H}$.

By construction, the quantity

$\displaystyle w_1 h(Y_1) + \dots + w_k h(Y_k)$

is maximised when ${(Y_1,\dots,Y_k) = (X_1,\dots,X_k)}$. At this point we could use the method of Lagrange multipliers to obtain the required constraints, but because we have some boundary conditions on the ${(Y_1,\dots,Y_k)}$ (namely, that the probability that they attain a given element of ${\Gamma}$ has to be non-negative) we will work things out by hand. Let ${t = (t_1,\dots,t_k)}$ be an element of ${\Gamma}$, and ${s = (s_1,\dots,s_k)}$ an element of ${\Gamma^*}$. For ${\varepsilon>0}$ small enough, we can form a random variable ${(Y_1,\dots,Y_k)}$ taking values in ${\Gamma}$, whose probability distribution is the same as that for ${(X_1,\dots,X_k)}$ except that the probability of attaining ${(t_1,\dots,t_k)}$ is increased by ${\varepsilon}$, and the probability of attaining ${(s_1,\dots,s_k)}$ is decreased by ${\varepsilon}$. If there is any ${j}$ for which ${{\bf P}(X_j = t_j)=0}$ and ${w_j \neq 0}$, then one can check that

$\displaystyle w_1 h(Y_1) + \dots + w_k h(Y_k) - (w_1 h(X_1) + \dots + w_k h(X_k)) \gg \varepsilon \log \frac{1}{\varepsilon}$

for sufficiently small ${\varepsilon}$, contradicting the maximality of ${(X_1,\dots,X_k)}$; thus we have ${{\bf P}(X_j = t_j) > 0}$ whenever ${w_j \neq 0}$. Taylor expansion then gives

$\displaystyle w_1 h(Y_1) + \dots + w_k h(Y_k) - (w_1 h(X_1) + \dots + w_k h(X_k)) = (A_t - A_s) \varepsilon + O(\varepsilon^2)$

for small ${\varepsilon}$, where

$\displaystyle A_t := \sum_{j=1}^k w_j \log \frac{1}{{\bf P}(X_j = t_j)}$

and similarly for ${A_s}$. We conclude that ${A_t \leq A_s}$ for all ${s \in \Gamma^*}$ and ${t \in \Gamma}$, thus there exists a quantity ${D}$ such that ${A_s = D}$ for all ${s \in \Gamma^*}$, and ${A_t \leq D}$ for all ${t \in \Gamma}$. By construction ${D}$ must be nonnegative. Sampling ${(t_1,\dots,t_k)}$ using the distribution of ${(X_1,\dots,X_k)}$, one has

$\displaystyle \sum_{j=1}^k w_j \log \frac{1}{{\bf P}(X_j = t_j)} = D$

almost surely; taking expectations we conclude that

$\displaystyle \sum_{j=1}^k w_j \sum_{t_j \in S_j} {\bf P}( X_j = t_j) \log \frac{1}{{\bf P}(X_j = t_j)} = D.$

The inner sum is ${h(X_j)}$, which equals ${H}$ when ${w_j}$ is non-zero, giving (17).

Now we show conversely that (ii) implies (i). As noted previously, the function ${p \mapsto p \log \frac{1}{p}}$ is concave on ${[0,1]}$, with derivative ${\log \frac{1}{p} - 1}$. This gives the inequality

$\displaystyle q \log \frac{1}{q} \leq p \log \frac{1}{p} + (q-p) ( \log \frac{1}{p} - 1 ) \ \ \ \ \ (18)$

for any ${0 \leq p,q \leq 1}$ (note the right-hand side may be infinite when ${p=0}$ and ${q>0}$). Let ${(Y_1,\dots,Y_k)}$ be any random variable taking values in ${\Gamma}$, then on applying the above inequality with ${p = {\bf P}(X_j = t_j)}$ and ${q = {\bf P}( Y_j = t_j )}$, multiplying by ${w_j}$, and summing over ${j=1,\dots,k}$ and ${t_j \in S_j}$ gives

$\displaystyle \sum_{j=1}^k w_j h(Y_j) \leq \sum_{j=1}^k w_j h(X_j)$

$\displaystyle + \sum_{j=1}^k \sum_{t_j \in S_j} w_j ({\bf P}(Y_j = t_j) - {\bf P}(X_j = t_j)) ( \log \frac{1}{{\bf P}(X_j=t_j)} - 1 ).$

By construction, one has

$\displaystyle \sum_{j=1}^k w_j h(X_j) = \min(h(X_1),\dots,h(X_k)) \sum_{j=1}^k w_j$

and

$\displaystyle \sum_{j=1}^k w_j h(Y_j) \geq \min(h(Y_1),\dots,h(Y_k)) \sum_{j=1}^k w_j$

so to prove that ${\min(h(Y_1),\dots,h(Y_k)) \leq \min(h(X_1),\dots,h(X_k))}$ (which would give (i)), it suffices to show that

$\displaystyle \sum_{j=1}^k \sum_{t_j \in S_j} w_j ({\bf P}(Y_j = t_j) - {\bf P}(X_j = t_j)) ( \log \frac{1}{{\bf P}(X_j=t_j)} - 1 ) \leq 0,$

or equivalently that the quantity

$\displaystyle \sum_{j=1}^k \sum_{t_j \in S_j} w_j {\bf P}(Y_j = t_j) ( \log \frac{1}{{\bf P}(X_j=t_j)} - 1 )$

is maximised when ${(Y_1,\dots,Y_k) = (X_1,\dots,X_k)}$. Since

$\displaystyle \sum_{j=1}^k \sum_{t_j \in S_j} w_j {\bf P}(Y_j = t_j) = \sum_{j=1}^k w_j$

it suffices to show this claim for the quantity

$\displaystyle \sum_{j=1}^k \sum_{t_j \in S_j} w_j {\bf P}(Y_j = t_j) \log \frac{1}{{\bf P}(X_j=t_j)}.$

One can view this quantity as

$\displaystyle {\bf E}_{(Y_1,\dots,Y_k)} \sum_{j=1}^k w_j \log \frac{1}{{\bf P}_{X_j}(X_j=Y_j)}.$

By (ii), this quantity is bounded by ${D}$, with equality if ${(Y_1,\dots,Y_k)}$ is equal to ${(X_1,\dots,X_k)}$ (and is in particular ranging in ${\Gamma^*}$), giving the claim. $\Box$

The second half of the proof of Proposition 7 only uses the marginal distributions ${{{\bf P}(X_j=t_j)}}$ and the equation(16), not the actual distribution of ${(X_1,\dots,X_k)}$, so it can also be used to prove an upper bound on ${H}$ when the exact maximizing distribution is not known, given suitable probability distributions in each variable. The logarithm of the probability distribution here plays the role that the weight functions do in BCCGNSU.

Remark 8 Suppose one is in the situation of (i) and (ii) above; assume the nondegeneracy condition that ${H}$ is positive (or equivalently that ${D}$ is positive). We can assign a “degree” ${d_j(t_j)}$ to each element ${t_j \in S_j}$ by the formula

$\displaystyle d_j(t_j) := w_j \log \frac{1}{{\bf P}(X_j = t_j)}, \ \ \ \ \ (19)$

then every tuple ${(t_1,\dots,t_k)}$ in ${\Gamma}$ has total degree at most ${D}$, and those tuples in ${\Gamma^*}$ have degree exactly ${D}$. In particular, every tuple in ${\Gamma^n}$ has degree at most ${nD}$, and hence by (17), each such tuple has a ${j}$-component of degree less than or equal to ${nHw_j}$ for some ${j}$ with ${w_j>0}$. On the other hand, we can compute from (19) and the fact that ${h(X_j) = H}$ for ${w_j > 0}$ that ${Hw_j = {\bf E} d_j(X_j)}$. Thus, by asymptotic equipartition, and assuming ${w_j \neq 0}$, the number of “monomials” in ${S_j^n}$ of total degree at most ${nHw_j}$ is at most ${\exp( (h(X_j)+o(1)) n )}$; one can in fact use (19) and (18) to show that this is in fact an equality. This gives a direct way to cover ${\Gamma^n}$ by sets ${\Gamma_{n,1},\dots,\Gamma_{n,k}}$ with ${|\pi_j(\Gamma_{n,j})| \leq \exp( (H+o(1)) n)}$, which is in the spirit of the Croot-Lev-Pach-Ellenberg-Gijswijt arguments from the previous post.

We can now show that the rank computation for the capset problem is sharp:

Proposition 9 Let ${V_1^{\otimes n} = V_2^{\otimes n} = V_3^{\otimes n}}$ denote the space of functions from ${{\bf F}_3^n}$ to ${{\bf F}_3}$. Then the function ${(x,y,z) \mapsto \delta_{0^n}(x,y,z)}$ from ${{\bf F}_3^n \times {\bf F}_3^n \times {\bf F}_3^n}$ to ${{\bf F}}$, viewed as an element of ${V_1^{\otimes n} \otimes V_2^{\otimes n} \otimes V_3^{\otimes n}}$, has rank ${\exp( (H^*+o(1)) n )}$ as ${n \rightarrow \infty}$, where ${H^* \approx 1.013445}$ is given by the formula

$\displaystyle H^* = \alpha \log \frac{1}{\alpha} + \beta \log \frac{1}{\beta} + \gamma \log \frac{1}{\gamma} \ \ \ \ \ (20)$

with

$\displaystyle \alpha = \frac{32}{3(15 + \sqrt{33})} \approx 0.51419$

$\displaystyle \beta = \frac{4(\sqrt{33}-1)}{3(15+\sqrt{33})} \approx 0.30495$

$\displaystyle \gamma = \frac{(\sqrt{33}-1)^2}{6(15+\sqrt{33})} \approx 0.18086.$

Proof: In ${{\bf F}_3 \times {\bf F}_3 \times {\bf F}_3}$, we have

$\displaystyle \delta_0(x+y+z) = 1 - (x+y+z)^2$

$\displaystyle = (1-x^2) - y^2 - z^2 + xy + yz + zx.$

Thus, if we let ${V_1=V_2=V_3}$ be the space of functions from ${{\bf F}_3}$ to ${{\bf F}_3}$ (with domain variable denoted ${x,y,z}$ respectively), and define the basis functions

$\displaystyle v_{1,0} := 1; v_{1,1} := x; v_{1,2} := x^2$

$\displaystyle v_{2,0} := 1; v_{2,1} := y; v_{2,2} := y^2$

$\displaystyle v_{3,0} := 1; v_{3,1} := z; v_{3,2} := z^2$

of ${V_1,V_2,V_3}$ indexed by ${S_1=S_2=S_3 := \{ 0,1,2\}}$ (with the usual ordering), respectively, and set ${\Gamma \subset S_1 \times S_2 \times S_3}$ to be the set

$\displaystyle \{ (2,0,0), (0,2,0), (0,0,2), (1,1,0), (0,1,1), (1,0,1),(0,0,0) \}$

then ${\delta_0(x,y,z)}$ is a linear combination of the ${v_{1,t_1} \otimes v_{1,t_2} \otimes v_{1,t_3}}$ with ${(t_1,t_2,t_3) \in \Gamma}$, and all coefficients non-zero. Then we have ${\Gamma'= \{ (2,0,0), (0,2,0), (0,0,2), (1,1,0), (0,1,1), (1,0,1) \}}$. We will show that the quantity ${H}$ of (10) agrees with the quantity ${H^*}$ of (20), and that the optimizing distribution is supported on ${\Gamma'}$, so that by Proposition 6 the rank of ${\delta_{0^n}(x,y,z)}$ is ${\exp( (H+o(1)) n)}$.

To compute the quantity at (10), we use the criterion in Proposition 7. We take ${(X_1,X_2,X_3)}$ to be the random variable taking values in ${\Gamma}$ that attains each of the values ${(2,0,0), (0,2,0), (0,0,2)}$ with a probability of ${\gamma \approx 0.18086}$, and each of ${(1,1,0), (0,1,1), (1,0,1)}$ with a probability of ${\alpha - 2\gamma = \beta/2 \approx 0.15247}$; then each of the ${X_j}$ attains the values of ${0,1,2}$ with probabilities ${\alpha,\beta,\gamma}$ respectively, so in particular ${h(X_1)=h(X_2)=h(X_3)}$ is equal to the quantity ${H'}$ in (20). If we now set ${w_1 = w_2 = w_3 := 1}$ and

$\displaystyle D := 2\log \frac{1}{\alpha} + \log \frac{1}{\gamma} = \log \frac{1}{\alpha} + 2 \log \frac{1}{\beta} = 3H^* \approx 3.04036$

we can verify the condition (16) with equality for all ${(t_1,t_2,t_3) \in \Gamma'}$, which from (17) gives ${H=H'=H^*}$ as desired. $\Box$

This statement already follows from the result of Kleinberg-Sawin-Speyer, which gives a “tri-colored sum-free set” in ${\mathbb F_3^n}$ of size ${\exp((H'+o(1))n)}$, as the slice rank of this tensor is an upper bound for the size of a tri-colored sum-free set. If one were to go over the proofs more carefully to evaluate the subexponential factors, this argument would give a stronger lower bound than KSS, as it does not deal with the substantial loss that comes from Behrend’s construction. However, because it actually constructs a set, the KSS result rules out more possible approaches to give an exponential improvement of the upper bound for capsets. The lower bound on slice rank shows that the bound cannot be improved using only the slice rank of this particular tensor, whereas KSS shows that the bound cannot be improved using any method that does not take advantage of the “single-colored” nature of the problem.

We can also show that the slice rank upper bound in a result of Naslund-Sawin is similarly sharp:

Proposition 10 Let ${V_1^{\otimes n} = V_2^{\otimes n} = V_3^{\otimes n}}$ denote the space of functions from ${\{0,1\}^n}$ to ${\mathbb C}$. Then the function ${(x,y,z) \mapsto \prod_{i=1}^n (x_i+y_i+z_i)-1}$ from ${\{0,1\}^n \times \{0,1\}^n \times \{0,1\}^n \rightarrow \mathbb C}$, viewed as an element of ${V_1^{\otimes n} \otimes V_2^{\otimes n} \otimes V_3^{\otimes n}}$, has slice rank ${(3/2^{2/3})^n e^{o(n)}}$

Proof: Let ${v_{1,0}=1}$ and ${v_{1,1}=x}$ be a basis for the space ${V_1}$ of functions on ${\{0,1\}}$, itself indexed by ${S_1=\{0,1\}}$. Choose similar bases for ${V_2}$ and ${V_3}$, with ${v_{2,0}=1, v_{2,1}=y}$ and ${v_{3,0}=1,v_{3,1}=z-1}$.

Set ${\Gamma = \{(1,0,0),(0,1,0),(0,0,1)\}}$. Then ${x+y+z-1}$ is a linear combination of the ${v_{1,t_1} \otimes v_{1,t_2} \otimes v_{1,t_3}}$ with ${(t_1,t_2,t_3) \in \Gamma}$, and all coefficients non-zero. Order ${S_1,S_2,S_3}$ the usual way so that ${\Gamma}$ is an antichain. We will show that the quantity ${H}$ of (10) is ${\log(3/2^{2/3})}$, so that applying the last statement of Proposition 6, we conclude that the rank of ${\delta_{0^n}(x,y,z)}$ is ${\exp( (\log(3/2^{2/3})+o(1)) n)= (3/2^{2/3})^n e^{o(n)}}$ ,

Let ${(X_1,X_2,X_3)}$ be the random variable taking values in ${\Gamma}$ that attains each of the values ${(1,0,0),(0,1,0),(0,0,1)}$ with a probability of ${1/3}$. Then each of the ${X_i}$ attains the value ${1}$ with probability ${1/3}$ and ${0}$ with probability ${2/3}$, so

$\displaystyle h(X_1)=h(X_2)=h(X_3) = (1/3) \log (3) + (2/3) \log(3/2) = \log 3 - (2/3) \log 2= \log (3/2^{2/3})$

Setting ${w_1=w_2=w_3=1}$ and ${D=3 \log(3/2^{2/3})=3 \log 3 - 2 \log 2}$, we can verify the condition (16) with equality for all ${(t_1,t_2,t_3) \in \Gamma'}$, which from (17) gives ${H=\log (3/2^{2/3})}$ as desired. $\Box$

We used a slightly different method in each of the last two results. In the first one, we use the most natural bases for all three vector spaces, and distinguish ${\Gamma}$ from its set of maximal elements ${\Gamma'}$. In the second one we modify one basis element slightly, with ${v_{3,1}=z-1}$ instead of the more obvious choice ${z}$, which allows us to work with ${\Gamma = \{(1,0,0),(0,1,0),(0,0,1)\}}$ instead of ${\Gamma=\{(1,0,0),(0,1,0),(0,0,1),(0,0,0)\}}$. Because ${\Gamma}$ is an antichain, we do not need to distinguish ${\Gamma}$ and ${\Gamma'}$. Both methods in fact work with either problem, and they are both about equally difficult, but we include both as either might turn out to be substantially more convenient in future work.

Proposition 11 Let ${k \geq 8}$ be a natural number and let ${G}$ be a finite abelian group. Let ${{\bf F}}$ be any field. Let ${V_1 = \dots = V_k}$ denote the space of functions from ${G}$ to ${{\bf F}}$.

Let ${F}$ be any ${{\bf F}}$-valued function on ${G^k}$ that is nonzero only when the ${k}$ elements of ${G^n}$ form a ${k}$-term arithmetic progression, and is nonzero on every ${k}$-term constant progression.

Then the slice rank of ${F}$ is ${|G|}$.

Proof: We apply Proposition 4, using the standard bases of ${V_1,\dots,V_k}$. Let ${\Gamma}$ be the support of ${F}$. Suppose that we have ${k}$ orderings on ${H}$ such that the constant progressions are maximal elements of ${\Gamma}$ and thus all constant progressions lie in ${\Gamma'}$. Then for any partition ${\Gamma_1,\dots, \Gamma_k}$ of ${\Gamma'}$, ${\Gamma_j}$ can contain at most ${|\pi_j(\Gamma_j)|}$ constant progressions, and as all ${|G|}$ constant progressions must lie in one of the ${\Gamma_j}$, we must have ${\sum_{j=1}^k |\pi_j(\Gamma_j)| \geq |G|}$. By Proposition 4, this implies that the slice rank of ${F}$ is at least ${|G|}$. Since ${F}$ is a ${|G| \times \dots \times |G|}$ tensor, the slice rank is at most ${|G|}$, hence exactly ${|G|}$.

So it is sufficient to find ${k}$ orderings on ${G}$ such that the constant progressions are maximal element of ${\Gamma}$. We make several simplifying reductions: We may as well assume that ${\Gamma}$ consists of all the ${k}$-term arithmetic progressions, because if the constant progressions are maximal among the set of all progressions then they are maximal among its subset ${\Gamma}$. So we are looking for an ordering in which the constant progressions are maximal among all ${k}$-term arithmetic progressions. We may as well assume that ${G}$ is cyclic, because if for each cyclic group we have an ordering where constant progressions are maximal, on an arbitrary finite abelian group the lexicographic product of these orderings is an ordering for which the constant progressions are maximal. We may assume ${k=8}$, as if we have an ${8}$-tuple of orderings where constant progressions are maximal, we may add arbitrary orderings and the constant progressions will remain maximal.

So it is sufficient to find ${8}$ orderings on the cyclic group ${\mathbb Z/n}$ such that the constant progressions are maximal elements of the set of ${8}$-term progressions in ${\mathbb Z/n}$ in the ${8}$-fold product ordering. To do that, let the first, second, third, and fifth orderings be the usual order on ${\{0,\dots,n-1\}}$ and let the fourth, sixth, seventh, and eighth orderings be the reverse of the usual order on ${\{0,\dots,n-1\}}$.

Then let ${(c,c,c,c,c,c,c,c)}$ be a constant progression and for contradiction assume that ${(a,a+b,a+2b,a+3b,a+4b,a+5b,a+6b,a+7b)}$ is a progression greater than ${(c,c,c,c,c,c,c,c)}$ in this ordering. We may assume that ${c \in [0, (n-1)/2]}$, because otherwise we may reverse the order of the progression, which has the effect of reversing all eight orderings, and then apply the transformation ${x \rightarrow n-1-x}$, which again reverses the eight orderings, bringing us back to the original problem but with ${c \in [0,(n-1)/2]}$.

Take a representative of the residue class ${b}$ in the interval ${[-n/2,n/2]}$. We will abuse notation and call this ${b}$. Observe that ${a+b, a+2b,}$ ${a+3b}$, and ${a+5b}$ are all contained in the interval ${[0,c]}$ modulo ${n}$. Take a representative of the residue class ${a}$ in the interval ${[0,c]}$. Then ${a+b}$ is in the interval ${[mn,mn+c]}$ for some ${m}$. The distance between any distinct pair of intervals of this type is greater than ${n/2}$, but the distance between ${a}$ and ${a+b}$ is at most ${n/2}$, so ${a+b}$ is in the interval ${[0,c]}$. By the same reasoning, ${a+2b}$ is in the interval ${[0,c]}$. Therefore ${|b| \leq c/2< n/4}$. But then the distance between ${a+2b}$ and ${a+4b}$ is at most ${n/2}$, so by the same reasoning ${a+4b}$ is in the interval ${[0,c]}$. Because ${a+3b}$ is between ${a+2b}$ and ${a+4b}$, it also lies in the interval ${[0,c]}$. Because ${a+3b}$ is in the interval ${[0,c]}$, and by assumption it is congruent mod ${n}$ to a number in the set ${\{0,\dots,n-1\}}$ greater than or equal to ${c}$, it must be exactly ${c}$. Then, remembering that ${a+2b}$ and ${a+4b}$ lie in ${[0,c]}$, we have ${c-b \leq b}$ and ${c+b \leq b}$, so ${b=0}$, hence ${a=c}$, thus ${(a,\dots,a+7b)=(c,\dots,c)}$, which contradicts the assumption that ${(a,\dots,a+7b)>(c,\dots,c)}$. $\Box$

In fact, given a ${k}$-term progressions mod ${n}$ and a constant, we can form a ${k}$-term binary sequence with a ${1}$ for each step of the progression that is greater than the constant and a ${0}$ for each step that is less. Because a rotation map, viewed as a dynamical system, has zero topological entropy, the number of ${k}$-term binary sequences that appear grows subexponentially in ${k}$. Hence there must be, for large enough ${k}$, at least one sequence that does not appear. In this proof we exploit a sequence that does not appear for ${k=8}$.

A capset in the vector space ${{\bf F}_3^n}$ over the finite field ${{\bf F}_3}$ of three elements is a subset ${A}$ of ${{\bf F}_3^n}$ that does not contain any lines ${\{ x,x+r,x+2r\}}$, where ${x,r \in {\bf F}_3^n}$ and ${r \neq 0}$. A basic problem in additive combinatorics (discussed in one of the very first posts on this blog) is to obtain good upper and lower bounds for the maximal size of a capset in ${{\bf F}_3^n}$.

Trivially, one has ${|A| \leq 3^n}$. Using Fourier methods (and the density increment argument of Roth), the bound of ${|A| \leq O( 3^n / n )}$ was obtained by Meshulam, and improved only as late as 2012 to ${O( 3^n /n^{1+c})}$ for some absolute constant ${c>0}$ by Bateman and Katz. But in a very recent breakthrough, Ellenberg (and independently Gijswijt) obtained the exponentially superior bound ${|A| \leq O( 2.756^n )}$, using a version of the polynomial method recently introduced by Croot, Lev, and Pach. (In the converse direction, a construction of Edel gives capsets as large as ${(2.2174)^n}$.) Given the success of the polynomial method in superficially similar problems such as the finite field Kakeya problem (discussed in this previous post), it was natural to wonder that this method could be applicable to the cap set problem (see for instance this MathOverflow comment of mine on this from 2010), but it took a surprisingly long time before Croot, Lev, and Pach were able to identify the precise variant of the polynomial method that would actually work here.

The proof of the capset bound is very short (Ellenberg’s and Gijswijt’s preprints are both 3 pages long, and Croot-Lev-Pach is 6 pages), but I thought I would present a slight reformulation of the argument which treats the three points on a line in ${{\bf F}_3}$ symmetrically (as opposed to treating the third point differently from the first two, as is done in the Ellenberg and Gijswijt papers; Croot-Lev-Pach also treat the middle point of a three-term arithmetic progression differently from the two endpoints, although this is a very natural thing to do in their context of ${({\bf Z}/4{\bf Z})^n}$). The basic starting point is this: if ${A}$ is a capset, then one has the identity

$\displaystyle \delta_{0^n}( x+y+z ) = \sum_{a \in A} \delta_a(x) \delta_a(y) \delta_a(z) \ \ \ \ \ (1)$

for all ${(x,y,z) \in A^3}$, where ${\delta_a(x) := 1_{a=x}}$ is the Kronecker delta function, which we view as taking values in ${{\bf F}_3}$. Indeed, (1) reflects the fact that the equation ${x+y+z=0}$ has solutions precisely when ${x,y,z}$ are either all equal, or form a line, and the latter is ruled out precisely when ${A}$ is a capset.

To exploit (1), we will show that the left-hand side of (1) is “low rank” in some sense, while the right-hand side is “high rank”. Recall that a function ${F: A \times A \rightarrow {\bf F}}$ taking values in a field ${{\bf F}}$ is of rank one if it is non-zero and of the form ${(x,y) \mapsto f(x) g(y)}$ for some ${f,g: A \rightarrow {\bf F}}$, and that the rank of a general function ${F: A \times A \rightarrow {\bf F}}$ is the least number of rank one functions needed to express ${F}$ as a linear combination. More generally, if ${k \geq 2}$, we define the rank of a function ${F: A^k \rightarrow {\bf F}}$ to be the least number of “rank one” functions of the form

$\displaystyle (x_1,\dots,x_k) \mapsto f(x_i) g(x_1,\dots,x_{i-1},x_{i+1},\dots,x_k)$

for some ${i=1,\dots,k}$ and some functions ${f: A \rightarrow {\bf F}}$, ${g: A^{k-1} \rightarrow {\bf F}}$, that are needed to generate ${F}$ as a linear combination. For instance, when ${k=3}$, the rank one functions take the form ${(x,y,z) \mapsto f(x) g(y,z)}$, ${(x,y,z) \mapsto f(y) g(x,z)}$, ${(x,y,z) \mapsto f(z) g(x,y)}$, and linear combinations of ${r}$ such rank one functions will give a function of rank at most ${r}$.

It is a standard fact in linear algebra that the rank of a diagonal matrix is equal to the number of non-zero entries. This phenomenon extends to higher dimensions:

Lemma 1 (Rank of diagonal hypermatrices) Let ${k \geq 2}$, let ${A}$ be a finite set, let ${{\bf F}}$ be a field, and for each ${a \in A}$, let ${c_a \in {\bf F}}$ be a coefficient. Then the rank of the function

$\displaystyle (x_1,\dots,x_k) \mapsto \sum_{a \in A} c_a \delta_a(x_1) \dots \delta_a(x_k) \ \ \ \ \ (2)$

is equal to the number of non-zero coefficients ${c_a}$.

Proof: We induct on ${k}$. As mentioned above, the case ${k=2}$ follows from standard linear algebra, so suppose now that ${k>2}$ and the claim has already been proven for ${k-1}$.

It is clear that the function (2) has rank at most equal to the number of non-zero ${c_a}$ (since the summands on the right-hand side are rank one functions), so it suffices to establish the lower bound. By deleting from ${A}$ those elements ${a \in A}$ with ${c_a=0}$ (which cannot increase the rank), we may assume without loss of generality that all the ${c_a}$ are non-zero. Now suppose for contradiction that (2) has rank at most ${|A|-1}$, then we obtain a representation

$\displaystyle \sum_{a \in A} c_a \delta_a(x_1) \dots \delta_a(x_k)$

$\displaystyle = \sum_{i=1}^k \sum_{\alpha \in I_i} f_{i,\alpha}(x_i) g_{i,\alpha}( x_1,\dots,x_{i-1},x_{i+1},\dots,x_k) \ \ \ \ \ (3)$

for some sets ${I_1,\dots,I_k}$ of cardinalities adding up to at most ${|A|-1}$, and some functions ${f_{i,\alpha}: A \rightarrow {\bf F}}$ and ${g_{i,\alpha}: A^{k-1} \rightarrow {\bf R}}$.

Consider the space of functions ${h: A \rightarrow {\bf F}}$ that are orthogonal to all the ${f_{k,\alpha}}$, ${\alpha \in I_k}$ in the sense that

$\displaystyle \sum_{x \in A} f_{k,\alpha}(x) h(x) = 0$

for all ${\alpha \in I_k}$. This space is a vector space whose dimension ${d}$ is at least ${|A| - |I_k|}$. A basis of this space generates a ${d \times |A|}$ coordinate matrix of full rank, which implies that there is at least one non-singular ${d \times d}$ minor. This implies that there exists a function ${h: A \rightarrow {\bf F}}$ in this space which is nowhere vanishing on some subset ${A'}$ of ${A}$ of cardinality at least ${|A|-|I_k|}$.

If we multiply (3) by ${h(x_k)}$ and sum in ${x_k}$, we conclude that

$\displaystyle \sum_{a \in A} c_a h(a) \delta_a(x_1) \dots \delta_a(x_{k-1})$

$\displaystyle = \sum_{i=1}^{k-1} \sum_{\alpha \in I_i} f_{i,\alpha}(x_i)\tilde g_{i,\alpha}( x_1,\dots,x_{i-1},x_{i+1},\dots,x_{k-1})$

where

$\displaystyle \tilde g_{i,\alpha}(x_1,\dots,x_{i-1},x_{i+1},\dots,x_{k-1})$

$\displaystyle := \sum_{x_k \in A} g_{i,\alpha}(x_1,\dots,x_{i-1},x_{i+1},\dots,x_k) h(x_k).$

The right-hand side has rank at most ${|A|-1-|I_k|}$, since the summands are rank one functions. On the other hand, from induction hypothesis the left-hand side has rank at least ${|A|-|I_k|}$, giving the required contradiction. $\Box$

On the other hand, we have the following (symmetrised version of a) beautifully simple observation of Croot, Lev, and Pach:

Lemma 2 On ${({\bf F}_3^n)^3}$, the rank of the function ${(x,y,z) \mapsto \delta_{0^n}(x+y+z)}$ is at most ${3N}$, where

$\displaystyle N := \sum_{a,b,c \geq 0: a+b+c=n, b+2c \leq 2n/3} \frac{n!}{a!b!c!}.$

Proof: Using the identity ${\delta_0(x) = 1 - x^2}$ for ${x \in {\bf F}_3}$, we have

$\displaystyle \delta_{0^n}(x+y+z) = \prod_{i=1}^n (1 - (x_i+y_i+z_i)^2).$

The right-hand side is clearly a polynomial of degree ${2n}$ in ${x,y,z}$, which is then a linear combination of monomials

$\displaystyle x_1^{i_1} \dots x_n^{i_n} y_1^{j_1} \dots y_n^{j_n} z_1^{k_1} \dots z_n^{k_n}$

with ${i_1,\dots,i_n,j_1,\dots,j_n,k_1,\dots,k_n \in \{0,1,2\}}$ with

$\displaystyle i_1 + \dots + i_n + j_1 + \dots + j_n + k_1 + \dots + k_n \leq 2n.$

In particular, from the pigeonhole principle, at least one of ${i_1 + \dots + i_n, j_1 + \dots + j_n, k_1 + \dots + k_n}$ is at most ${2n/3}$.

Consider the contribution of the monomials for which ${i_1 + \dots + i_n \leq 2n/3}$. We can regroup this contribution as

$\displaystyle \sum_\alpha f_\alpha(x) g_\alpha(y,z)$

where ${\alpha}$ ranges over those ${(i_1,\dots,i_n) \in \{0,1,2\}^n}$ with ${i_1 + \dots + i_n \leq 2n/3}$, ${f_\alpha}$ is the monomial

$\displaystyle f_\alpha(x_1,\dots,x_n) := x_1^{i_1} \dots x_n^{i_n}$

and ${g_\alpha: {\bf F}_3^n \times {\bf F}_3^n \rightarrow {\bf F}_3}$ is some explicitly computable function whose exact form will not be of relevance to our argument. The number of such ${\alpha}$ is equal to ${N}$, so this contribution has rank at most ${N}$. The remaining contributions arising from the cases ${j_1 + \dots + j_n \leq 2n/3}$ and ${k_1 + \dots + k_n \leq 2n/3}$ similarly have rank at most ${N}$ (grouping the monomials so that each monomial is only counted once), so the claim follows.

Upon restricting from ${({\bf F}_3^n)^3}$ to ${A^3}$, the rank of ${(x,y,z) \mapsto \delta_{0^n}(x+y+z)}$ is still at most ${3N}$. The two lemmas then combine to give the Ellenberg-Gijswijt bound

$\displaystyle |A| \leq 3N.$

All that remains is to compute the asymptotic behaviour of ${N}$. This can be done using the general tool of Cramer’s theorem, but can also be derived from Stirling’s formula (discussed in this previous post). Indeed, if ${a = (\alpha+o(1)) n}$, ${b = (\beta+o(1)) n}$, ${c = (\gamma+o(1)) n}$ for some ${\alpha,\beta,\gamma \geq 0}$ summing to ${1}$, Stirling’s formula gives

$\displaystyle \frac{n!}{a!b!c!} = \exp( n (h(\alpha,\beta,\gamma) + o(1)) )$

where ${h}$ is the entropy function

$\displaystyle h(\alpha,\beta,\gamma) = \alpha \log \frac{1}{\alpha} + \beta \log \frac{1}{\beta} + \gamma \log \frac{1}{\gamma}.$

We then have

$\displaystyle N = \exp( n (X + o(1))$

where ${X}$ is the maximum entropy ${h(\alpha,\beta,\gamma)}$ subject to the constraints

$\displaystyle \alpha,\beta,\gamma \geq 0; \alpha+\beta+\gamma=1; \beta+2\gamma \leq 2/3.$

A routine Lagrange multiplier computation shows that the maximum occurs when

$\displaystyle \alpha = \frac{32}{3(15 + \sqrt{33})}$

$\displaystyle \beta = \frac{4(\sqrt{33}-1)}{3(15+\sqrt{33})}$

$\displaystyle \gamma = \frac{(\sqrt{33}-1)^2}{6(15+\sqrt{33})}$

and ${h(\alpha,\beta,\gamma)}$ is approximately ${1.013455}$, giving rise to the claimed bound of ${O( 2.756^n )}$.

Remark 3 As noted in the Ellenberg and Gijswijt papers, the above argument extends readily to other fields than ${{\bf F}_3}$ to control the maximal size of subset of ${{\bf F}^n}$ that has no non-trivial solutions to the equation ${ax+by+cz=0}$, where ${a,b,c \in {\bf F}}$ are non-zero constants that sum to zero. Of course one replaces the function ${(x,y,z) \mapsto \delta_{0^n}(x+y+z)}$ in Lemma 2 by ${(x,y,z) \mapsto \delta_{0^n}(ax+by+cz)}$ in this case.

Remark 4 This symmetrised formulation suggests that one possible way to improve slightly on the numerical quantity ${2.756}$ by finding a more efficient way to decompose ${\delta_{0^n}(x+y+z)}$ into rank one functions, however I was not able to do so (though such improvements are reminiscent of the Strassen type algorithms for fast matrix multiplication).

Remark 5 It is tempting to see if this method can get non-trivial upper bounds for sets ${A}$ with no length ${4}$ progressions, in (say) ${{\bf F}_5^n}$. One can run the above arguments, replacing the function

$\displaystyle (x,y,z) \mapsto \delta_{0^n}(x+y+z)$

with

$\displaystyle (x,y,z,w) \mapsto \delta_{0^n}(x-2y+z) \delta_{0^n}(y-2z+w);$

this leads to the bound ${|A| \leq 4N}$ where

$\displaystyle N := \sum_{a,b,c,d,e \geq 0: a+b+c+d+e=n, b+2c+3d+4e \leq 2n} \frac{n!}{a!b!c!d!e!}.$

Unfortunately, ${N}$ is asymptotic to ${\frac{1}{2} 5^n}$ and so this bound is in fact slightly worse than the trivial bound ${|A| \leq 5^n}$! However, there is a slim chance that there is a more efficient way to decompose ${\delta_{0^n}(x-2y+z) \delta_{0^n}(y-2z+w)}$ into rank one functions that would give a non-trivial bound on ${A}$. I experimented with a few possible such decompositions but unfortunately without success.

Remark 6 Return now to the capset problem. Since Lemma 1 is valid for any field ${{\bf F}}$, one could perhaps hope to get better bounds by viewing the Kronecker delta function ${\delta}$ as taking values in another field than ${{\bf F}_3}$, such as the complex numbers ${{\bf C}}$. However, as soon as one works in a field of characteristic other than ${3}$, one can adjoin a cube root ${\omega}$ of unity, and one now has the Fourier decomposition

$\displaystyle \delta_{0^n}(x+y+z) = \frac{1}{3^n} \sum_{\xi \in {\bf F}_3^n} \omega^{\xi \cdot x} \omega^{\xi \cdot y} \omega^{\xi \cdot z}.$

Moving to the Fourier basis, we conclude from Lemma 1 that the function ${(x,y,z) \mapsto \delta_{0^n}(x+y+z)}$ on ${{\bf F}_3^n}$ now has rank exactly ${3^n}$, and so one cannot improve upon the trivial bound of ${|A| \leq 3^n}$ by this method using fields of characteristic other than three as the range field. So it seems one has to stick with ${{\bf F}_3}$ (or the algebraic completion thereof).

Thanks to Jordan Ellenberg and Ben Green for helpful discussions.

Tamar Ziegler and I have just uploaded to the arXiv two related papers: “Concatenation theorems for anti-Gowers-uniform functions and Host-Kra characteoristic factors” and “polynomial patterns in primes“, with the former developing a “quantitative Bessel inequality” for local Gowers norms that is crucial in the latter.

We use the term “concatenation theorem” to denote results in which structural control of a function in two or more “directions” can be “concatenated” into structural control in a joint direction. A trivial example of such a concatenation theorem is the following: if a function ${f: {\bf Z} \times {\bf Z} \rightarrow {\bf R}}$ is constant in the first variable (thus ${x \mapsto f(x,y)}$ is constant for each ${y}$), and also constant in the second variable (thus ${y \mapsto f(x,y)}$ is constant for each ${x}$), then it is constant in the joint variable ${(x,y)}$. A slightly less trivial example: if a function ${f: {\bf Z} \times {\bf Z} \rightarrow {\bf R}}$ is affine-linear in the first variable (thus, for each ${y}$, there exist ${\alpha(y), \beta(y)}$ such that ${f(x,y) = \alpha(y) x + \beta(y)}$ for all ${x}$) and affine-linear in the second variable (thus, for each ${x}$, there exist ${\gamma(x), \delta(x)}$ such that ${f(x,y) = \gamma(x)y + \delta(x)}$ for all ${y}$) then ${f}$ is a quadratic polynomial in ${x,y}$; in fact it must take the form

$\displaystyle f(x,y) = \epsilon xy + \zeta x + \eta y + \theta \ \ \ \ \ (1)$

for some real numbers ${\epsilon, \zeta, \eta, \theta}$. (This can be seen for instance by using the affine linearity in ${y}$ to show that the coefficients ${\alpha(y), \beta(y)}$ are also affine linear.)

The same phenomenon extends to higher degree polynomials. Given a function ${f: G \rightarrow K}$ from one additive group ${G}$ to another, we say that ${f}$ is of degree less than ${d}$ along a subgroup ${H}$ of ${G}$ if all the ${d}$-fold iterated differences of ${f}$ along directions in ${H}$ vanish, that is to say

$\displaystyle \partial_{h_1} \dots \partial_{h_d} f(x) = 0$

for all ${x \in G}$ and ${h_1,\dots,h_d \in H}$, where ${\partial_h}$ is the difference operator

$\displaystyle \partial_h f(x) := f(x+h) - f(x).$

(We adopt the convention that the only ${f}$ of degree less than ${0}$ is the zero function.)

We then have the following simple proposition:

Proposition 1 (Concatenation of polynomiality) Let ${f: G \rightarrow K}$ be of degree less than ${d_1}$ along one subgroup ${H_1}$ of ${G}$, and of degree less than ${d_2}$ along another subgroup ${H_2}$ of ${G}$, for some ${d_1,d_2 \geq 1}$. Then ${f}$ is of degree less than ${d_1+d_2-1}$ along the subgroup ${H_1+H_2}$ of ${G}$.

Note the previous example was basically the case when ${G = {\bf Z} \times {\bf Z}}$, ${H_1 = {\bf Z} \times \{0\}}$, ${H_2 = \{0\} \times {\bf Z}}$, ${K = {\bf R}}$, and ${d_1=d_2=2}$.

Proof: The claim is trivial for ${d_1=1}$ or ${d_2=1}$ (in which ${f}$ is constant along ${H_1}$ or ${H_2}$ respectively), so suppose inductively ${d_1,d_2 \geq 2}$ and the claim has already been proven for smaller values of ${d_1-1}$.

We take a derivative in a direction ${h_1 \in H_1}$ along ${h_1}$ to obtain

$\displaystyle T^{-h_1} f = f + \partial_{h_1} f$

where ${T^{-h_1} f(x) = f(x+h_1)}$ is the shift of ${f}$ by ${-h_1}$. Then we take a further shift by a direction ${h_2 \in H_2}$ to obtain

$\displaystyle T^{-h_1-h_2} f = T^{-h_2} f + T^{-h_2} \partial_{h_1} f = f + \partial_{h_2} f + T^{-h_2} \partial_{h_1} f$

$\displaystyle \partial_{h_1+h_2} f = \partial_{h_2} f + T^{-h_2} \partial_{h_1} f.$

Since ${f}$ has degree less than ${d_1}$ along ${H_1}$ and degree less than ${d_2}$ along ${H_2}$, ${\partial_{h_1} f}$ has degree less than ${d_1-1}$ along ${H_1}$ and less than ${d_2}$ along ${H_2}$, so is degree less than ${d_1+d_2-2}$ along ${H_1+H_2}$ by induction hypothesis. Similarly ${\partial_{h_2} f}$ is also of degree less than ${d_1+d_2-2}$ along ${H_1+H_2}$. Combining this with the cocycle equation we see that ${\partial_{h_1+h_2}f}$ is of degree less than ${d_1+d_2-2}$ along ${H_1+H_2}$ for any ${h_1+h_2 \in H_1+H_2}$, and hence ${f}$ is of degree less than ${d_1+d_2-1}$ along ${H_1+H_2}$, as required. $\Box$

While this proposition is simple, it already illustrates some basic principles regarding how one would go about proving a concatenation theorem:

• (i) One should perform induction on the degrees ${d_1,d_2}$ involved, and take advantage of the recursive nature of degree (in this case, the fact that a function is of less than degree ${d}$ along some subgroup ${H}$ of directions iff all of its first derivatives along ${H}$ are of degree less than ${d-1}$).
• (ii) Structure is preserved by operations such as addition, shifting, and taking derivatives. In particular, if a function ${f}$ is of degree less than ${d}$ along some subgroup ${H}$, then any derivative ${\partial_k f}$ of ${f}$ is also of degree less than ${d}$ along ${H}$, even if ${k}$ does not belong to ${H}$.

Here is another simple example of a concatenation theorem. Suppose an at most countable additive group ${G}$ acts by measure-preserving shifts ${T: g \mapsto T^g}$ on some probability space ${(X, {\mathcal X}, \mu)}$; we call the pair ${(X,T)}$ (or more precisely ${(X, {\mathcal X}, \mu, T)}$) a ${G}$-system. We say that a function ${f \in L^\infty(X)}$ is a generalised eigenfunction of degree less than ${d}$ along some subgroup ${H}$ of ${G}$ and some ${d \geq 1}$ if one has

$\displaystyle T^h f = \lambda_h f$

almost everywhere for all ${h \in H}$, and some functions ${\lambda_h \in L^\infty(X)}$ of degree less than ${d-1}$ along ${H}$, with the convention that a function has degree less than ${0}$ if and only if it is equal to ${1}$. Thus for instance, a function ${f}$ is an generalised eigenfunction of degree less than ${1}$ along ${H}$ if it is constant on almost every ${H}$-ergodic component of ${G}$, and is a generalised function of degree less than ${2}$ along ${H}$ if it is an eigenfunction of the shift action on almost every ${H}$-ergodic component of ${G}$. A basic example of a higher order eigenfunction is the function ${f(x,y) := e^{2\pi i y}}$ on the skew shift ${({\bf R}/{\bf Z})^2}$ with ${{\bf Z}}$ action given by the generator ${T(x,y) := (x+\alpha,y+x)}$ for some irrational ${\alpha}$. One can check that ${T^h f = \lambda_h f}$ for every integer ${h}$, where ${\lambda_h: x \mapsto e^{2\pi i \binom{h}{2} \alpha} e^{2\pi i h x}}$ is a generalised eigenfunction of degree less than ${2}$ along ${{\bf Z}}$, so ${f}$ is of degree less than ${3}$ along ${{\bf Z}}$.

We then have

Proposition 2 (Concatenation of higher order eigenfunctions) Let ${(X,T)}$ be a ${G}$-system, and let ${f \in L^\infty(X)}$ be a generalised eigenfunction of degree less than ${d_1}$ along one subgroup ${H_1}$ of ${G}$, and a generalised eigenfunction of degree less than ${d_2}$ along another subgroup ${H_2}$ of ${G}$, for some ${d_1,d_2 \geq 1}$. Then ${f}$ is a generalised eigenfunction of degree less than ${d_1+d_2-1}$ along the subgroup ${H_1+H_2}$ of ${G}$.

The argument is almost identical to that of the previous proposition and is left as an exercise to the reader. The key point is the point (ii) identified earlier: the space of generalised eigenfunctions of degree less than ${d}$ along ${H}$ is preserved by multiplication and shifts, as well as the operation of “taking derivatives” ${f \mapsto \lambda_k}$ even along directions ${k}$ that do not lie in ${H}$. (To prove this latter claim, one should restrict to the region where ${f}$ is non-zero, and then divide ${T^k f}$ by ${f}$ to locate ${\lambda_k}$.)

A typical example of this proposition in action is as follows: consider the ${{\bf Z}^2}$-system given by the ${3}$-torus ${({\bf R}/{\bf Z})^3}$ with generating shifts

$\displaystyle T^{(1,0)}(x,y,z) := (x+\alpha,y,z+y)$

$\displaystyle T^{(0,1)}(x,y,z) := (x,y+\alpha,z+x)$

for some irrational ${\alpha}$, which can be checked to give a ${{\bf Z}^2}$ action

$\displaystyle T^{(n,m)}(x,y,z) := (x+n\alpha, y+m\alpha, z+ny+mx+nm\alpha).$

The function ${f(x,y,z) := e^{2\pi i z}}$ can then be checked to be a generalised eigenfunction of degree less than ${2}$ along ${{\bf Z} \times \{0\}}$, and also less than ${2}$ along ${\{0\} \times {\bf Z}}$, and less than ${3}$ along ${{\bf Z}^2}$. One can view this example as the dynamical systems translation of the example (1) (see this previous post for some more discussion of this sort of correspondence).

The main results of our concatenation paper are analogues of these propositions concerning a more complicated notion of “polynomial-like” structure that are of importance in additive combinatorics and in ergodic theory. On the ergodic theory side, the notion of structure is captured by the Host-Kra characteristic factors ${Z^{ of a ${G}$-system ${X}$ along a subgroup ${H}$. These factors can be defined in a number of ways. One is by duality, using the Gowers-Host-Kra uniformity seminorms (defined for instance here) ${\| \|_{U^d_H(X)}}$. Namely, ${Z^{ is the factor of ${X}$ defined up to equivalence by the requirement that

$\displaystyle \|f\|_{U^d_H(X)} = 0 \iff {\bf E}(f | Z^{

An equivalent definition is in terms of the dual functions ${{\mathcal D}^d_H(f)}$ of ${f}$ along ${H}$, which can be defined recursively by setting ${{\mathcal D}^0_H(f) = 1}$ and

$\displaystyle {\mathcal D}^d_H(f) = {\bf E}_h T^h f {\mathcal D}^{d-1}( f \overline{T^h f} )$

where ${{\bf E}_h}$ denotes the ergodic average along a Følner sequence in ${G}$ (in fact one can also define these concepts in non-amenable abelian settings as per this previous post). The factor ${Z^{ can then be alternately defined as the factor generated by the dual functions ${{\mathcal D}^d_H(f)}$ for ${f \in L^\infty(X)}$.

In the case when ${G=H={\bf Z}}$ and ${X}$ is ${G}$-ergodic, a deep theorem of Host and Kra shows that the factor ${Z^{ is equivalent to the inverse limit of nilsystems of step less than ${d}$. A similar statement holds with ${{\bf Z}}$ replaced by any finitely generated group by Griesmer, while the case of an infinite vector space over a finite field was treated in this paper of Bergelson, Ziegler, and myself. The situation is more subtle when ${X}$ is not ${G}$-ergodic, or when ${X}$ is ${G}$-ergodic but ${H}$ is a proper subgroup of ${G}$ acting non-ergodically, when one has to start considering measurable families of directional nilsystems; see for instance this paper of Austin for some of the subtleties involved (for instance, higher order group cohomology begins to become relevant!).

One of our main theorems is then

Proposition 3 (Concatenation of characteristic factors) Let ${(X,T)}$ be a ${G}$-system, and let ${f}$ be measurable with respect to the factor ${Z^{ and with respect to the factor ${Z^{ for some ${d_1,d_2 \geq 1}$ and some subgroups ${H_1,H_2}$ of ${G}$. Then ${f}$ is also measurable with respect to the factor ${Z^{.

We give two proofs of this proposition in the paper; an ergodic-theoretic proof using the Host-Kra theory of “cocycles of type ${ (along a subgroup ${H}$)”, which can be used to inductively describe the factors ${Z^{, and a combinatorial proof based on a combinatorial analogue of this proposition which is harder to state (but which roughly speaking asserts that a function which is nearly orthogonal to all bounded functions of small ${U^{d_1}_{H_1}}$ norm, and also to all bounded functions of small ${U^{d_2}_{H_2}}$ norm, is also nearly orthogonal to alll bounded functions of small ${U^{d_1+d_2-1}_{H_1+H_2}}$ norm). The combinatorial proof parallels the proof of Proposition 2. A key point is that dual functions ${F := {\mathcal D}^d_H(f)}$ obey a property analogous to being a generalised eigenfunction, namely that

$\displaystyle T^h F = {\bf E}_k \lambda_{h,k} F_k$

where ${F_k := T^k F}$ and ${\lambda_{h,k} := {\mathcal D}^{d-1}( T^h f \overline{T^k f} )}$ is a “structured function of order ${d-1}$” along ${H}$. (In the language of this previous paper of mine, this is an assertion that dual functions are uniformly almost periodic of order ${d}$.) Again, the point (ii) above is crucial, and in particular it is key that any structure that ${F}$ has is inherited by the associated functions ${\lambda_{h,k}}$ and ${F_k}$. This sort of inheritance is quite easy to accomplish in the ergodic setting, as there is a ready-made language of factors to encapsulate the concept of structure, and the shift-invariance and ${\sigma}$-algebra properties of factors make it easy to show that just about any “natural” operation one performs on a function measurable with respect to a given factor, returns a function that is still measurable in that factor. In the finitary combinatorial setting, though, encoding the fact (ii) becomes a remarkably complicated notational nightmare, requiring a huge amount of “epsilon management” and “second-order epsilon management” (in which one manages not only scalar epsilons, but also function-valued epsilons that depend on other parameters). In order to avoid all this we were forced to utilise a nonstandard analysis framework for the combinatorial theorems, which made the arguments greatly resemble the ergodic arguments in many respects (though the two settings are still not equivalent, see this previous blog post for some comparisons between the two settings). Unfortunately the arguments are still rather complicated.

For combinatorial applications, dual formulations of the concatenation theorem are more useful. A direct dualisation of the theorem yields the following decomposition theorem: a bounded function which is small in ${U^{d_1+d_2-1}_{H_1+H_2}}$ norm can be split into a component that is small in ${U^{d_1}_{H_1}}$ norm, and a component that is small in ${U^{d_2}_{H_2}}$ norm. (One may wish to understand this type of result by first proving the following baby version: any function that has mean zero on every coset of ${H_1+H_2}$, can be decomposed as the sum of a function that has mean zero on every ${H_1}$ coset, and a function that has mean zero on every ${H_2}$ coset. This is dual to the assertion that a function that is constant on every ${H_1}$ coset and constant on every ${H_2}$ coset, is constant on every ${H_1+H_2}$ coset.) Combining this with some standard “almost orthogonality” arguments (i.e. Cauchy-Schwarz) give the following Bessel-type inequality: if one has a lot of subgroups ${H_1,\dots,H_k}$ and a bounded function is small in ${U^{2d-1}_{H_i+H_j}}$ norm for most ${i,j}$, then it is also small in ${U^d_{H_i}}$ norm for most ${i}$. (Here is a baby version one may wish to warm up on: if a function ${f}$ has small mean on ${({\bf Z}/p{\bf Z})^2}$ for some large prime ${p}$, then it has small mean on most of the cosets of most of the one-dimensional subgroups of ${({\bf Z}/p{\bf Z})^2}$.)

There is also a generalisation of the above Bessel inequality (as well as several of the other results mentioned above) in which the subgroups ${H_i}$ are replaced by more general coset progressions ${H_i+P_i}$ (of bounded rank), so that one has a Bessel inequailty controlling “local” Gowers uniformity norms such as ${U^d_{P_i}}$ by “global” Gowers uniformity norms such as ${U^{2d-1}_{P_i+P_j}}$. This turns out to be particularly useful when attempting to compute polynomial averages such as

$\displaystyle \sum_{n \leq N} \sum_{r \leq \sqrt{N}} f(n) g(n+r^2) h(n+2r^2) \ \ \ \ \ (2)$

for various functions ${f,g,h}$. After repeated use of the van der Corput lemma, one can control such averages by expressions such as

$\displaystyle \sum_{n \leq N} \sum_{h,m,k \leq \sqrt{N}} f(n) f(n+mh) f(n+mk) f(n+m(h+k))$

(actually one ends up with more complicated expressions than this, but let’s use this example for sake of discussion). This can be viewed as an average of various ${U^2}$ Gowers uniformity norms of ${f}$ along arithmetic progressions of the form ${\{ mh: h \leq \sqrt{N}\}}$ for various ${m \leq \sqrt{N}}$. Using the above Bessel inequality, this can be controlled in turn by an average of various ${U^3}$ Gowers uniformity norms along rank two generalised arithmetic progressions of the form ${\{ m_1 h_1 + m_2 h_2: h_1,h_2 \le \sqrt{N}\}}$ for various ${m_1,m_2 \leq \sqrt{N}}$. But for generic ${m_1,m_2}$, this rank two progression is close in a certain technical sense to the “global” interval ${\{ n: n \leq N \}}$ (this is ultimately due to the basic fact that two randomly chosen large integers are likely to be coprime, or at least have a small gcd). As a consequence, one can use the concatenation theorems from our first paper to control expressions such as (2) in terms of global Gowers uniformity norms. This is important in number theoretic applications, when one is interested in computing sums such as

$\displaystyle \sum_{n \leq N} \sum_{r \leq \sqrt{N}} \mu(n) \mu(n+r^2) \mu(n+2r^2)$

or

$\displaystyle \sum_{n \leq N} \sum_{r \leq \sqrt{N}} \Lambda(n) \Lambda(n+r^2) \Lambda(n+2r^2)$

where ${\mu}$ and ${\Lambda}$ are the Möbius and von Mangoldt functions respectively. This is because we are able to control global Gowers uniformity norms of such functions (thanks to results such as the proof of the inverse conjecture for the Gowers norms, the orthogonality of the Möbius function with nilsequences, and asymptotics for linear equations in primes), but much less control is currently available for local Gowers uniformity norms, even with the assistance of the generalised Riemann hypothesis (see this previous blog post for some further discussion).

By combining these tools and strategies with the “transference principle” approach from our previous paper (as improved using the recent “densification” technique of Conlon, Fox, and Zhao, discussed in this previous post), we are able in particular to establish the following result:

Theorem 4 (Polynomial patterns in the primes) Let ${P_1,\dots,P_k: {\bf Z} \rightarrow {\bf Z}}$ be polynomials of degree at most ${d}$, whose degree ${d}$ coefficients are all distinct, for some ${d \geq 1}$. Suppose that ${P_1,\dots,P_k}$ is admissible in the sense that for every prime ${p}$, there are ${n,r}$ such that ${n+P_1(r),\dots,n+P_k(r)}$ are all coprime to ${p}$. Then there exist infinitely many pairs ${n,r}$ of natural numbers such that ${n+P_1(r),\dots,n+P_k(r)}$ are prime.

Furthermore, we obtain an asymptotic for the number of such pairs ${n,r}$ in the range ${n \leq N}$, ${r \leq N^{1/d}}$ (actually for minor technical reasons we reduce the range of ${r}$ to be very slightly less than ${N^{1/d}}$). In fact one could in principle obtain asymptotics for smaller values of ${r}$, and relax the requirement that the degree ${d}$ coefficients be distinct with the requirement that no two of the ${P_i}$ differ by a constant, provided one had good enough local uniformity results for the Möbius or von Mangoldt functions. For instance, we can obtain an asymptotic for triplets of the form ${n, n+r,n+r^d}$ unconditionally for ${d \leq 5}$, and conditionally on GRH for all ${d}$, using known results on primes in short intervals on average.

The ${d=1}$ case of this theorem was obtained in a previous paper of myself and Ben Green (using the aforementioned conjectures on the Gowers uniformity norm and the orthogonality of the Möbius function with nilsequences, both of which are now proven). For higher ${d}$, an older result of Tamar and myself was able to tackle the case when ${P_1(0)=\dots=P_k(0)=0}$ (though our results there only give lower bounds on the number of pairs ${(n,r)}$, and no asymptotics). Both of these results generalise my older theorem with Ben Green on the primes containing arbitrarily long arithmetic progressions. The theorem also extends to multidimensional polynomials, in which case there are some additional previous results; see the paper for more details. We also get a technical refinement of our previous result on narrow polynomial progressions in (dense subsets of) the primes by making the progressions just a little bit narrower in the case of the density of the set one is using is small.

. This latter Bessel type inequality is particularly useful in combinatorial and number-theoretic applications, as it allows one to convert “global” Gowers uniformity norm (basically, bounds on norms such as ${U^{2d-1}_{H_i+H_j}}$) to “local” Gowers uniformity norm control.

Van Vu and I just posted to the arXiv our paper “sum-free sets in groups” (submitted to Discrete Analysis), as well as a companion survey article (submitted to J. Comb.). Given a subset ${A}$ of an additive group ${G = (G,+)}$, define the quantity ${\phi(A)}$ to be the cardinality of the largest subset ${B}$ of ${A}$ which is sum-free in ${A}$ in the sense that all the sums ${b_1+b_2}$ with ${b_1,b_2}$ distinct elements of ${B}$ lie outside of ${A}$. For instance, if ${A}$ is itself a group, then ${\phi(A)=1}$, since no two elements of ${A}$ can sum to something outside of ${A}$. More generally, if ${A}$ is the union of ${k}$ groups, then ${\phi(A)}$ is at most ${k}$, thanks to the pigeonhole principle.

If ${G}$ is the integers, then there are no non-trivial subgroups, and one can thus expect ${\phi(A)}$ to start growing with ${A}$. For instance, one has the following easy result:

Proposition 1 Let ${A}$ be a set of ${2^k}$ natural numbers. Then ${\phi(A) > k}$.

Proof: We use an argument of Ruzsa, which is based in turn on an older argument of Choi. Let ${x_1}$ be the largest element of ${A}$, and then recursively, once ${x_1,\dots,x_i}$ has been selected, let ${x_{i+1}}$ be the largest element of ${A}$ not equal to any of the ${x_1,\dots,x_i}$, such that ${x_{i+1}+x_j \not \in A}$ for all ${j=1,\dots,i}$, terminating this construction when no such ${x_{i+1}}$ can be located. This gives a sequence ${x_1 > x_2 > \dots > x_m}$ of elements in ${A}$ which are sum-free in ${A}$, and with the property that for any ${y \in A}$, either ${y}$ is equal to one of the ${x_i}$, or else ${y + x_i \in A}$ for some ${i}$ with ${x_i > y}$. Iterating this, we see that any ${y \in A}$ is of the form ${x_{i_1} - x_{i_2} - \dots - x_{i_j}}$ for some ${j \geq 1}$ and ${1 \leq i_1 < i_2 < \dots \leq i_j \leq m}$. The number of such expressions ${x_{i_1} - x_{i_2} - \dots - x_{i_j}}$ is at most ${2^{m}-1}$, thus ${2^k \leq 2^m-1}$ which implies ${m \geq k+1}$. Since ${\phi(A) \geq m}$, the claim follows. $\Box$

In particular, we have ${\phi(A) \gg \log |A|}$ for subsets ${A}$ of the integers. It has been possible to improve upon this easy bound, but only with remarkable effort. The best lower bound currently is

$\displaystyle \phi(A) \geq \log |A| (\log\log|A|)^{1/2 - o(1)},$

a result of Shao (building upon earlier work of Sudakov, Szemeredi, and Vu and of Dousse). In the opposite direction, a construction of Ruzsa gives examples of large sets ${A}$ with ${\phi(A) \leq \exp( O( \sqrt{\log |A|} ) )}$.

Using the standard tool of Freiman homomorphisms, the above results for the integers extend to other torsion-free abelian groups ${G}$. In our paper we study the opposite case where ${G}$ is finite (but still abelian). In this paper of Erdös (in which the quantity ${\phi(A)}$ was first introduced), the following question was posed: if ${A}$ is sufficiently large depending on ${\phi(A)}$, does this imply the existence of two elements ${x,y \in A}$ with ${x+y=0}$? As it turns out, we were able to find some simple counterexamples to this statement. For instance, if ${H}$ is any finite additive group, then the set ${A := \{ 1 \hbox{ mod } 7, 2 \hbox{ mod } 7, 4 \hbox{ mod } 7\} \times H \subset {\bf Z}/7{\bf Z} \times H}$ has ${\phi(A)=3}$ but with no ${x,y \in A}$ summing to zero; this type of example in fact works with ${7}$ replaced by any larger Mersenne prime, and we also have a counterexample in ${{\bf Z}/2^n{\bf Z}}$ for ${n}$ arbitrarily large. However, in the positive direction, we can show that the answer to Erdös’s question is positive if ${|G|}$ is assumed to have no small prime factors. That is to say,

Theorem 2 For every ${k \geq 1}$ there exists ${C \geq 1}$ such that if ${G}$ is a finite abelian group whose order is not divisible by any prime less than or equal to ${C}$, and ${A}$ is a subset of ${G}$ with order at least ${C}$ and ${\phi(A) \leq k}$, then there exist ${x,y \in A}$ with ${x+y=0}$.

There are two main tools used to prove this result. One is an “arithmetic removal lemma” proven by Král, Serra, and Vena. Note that the condition ${\phi(A) \leq k}$ means that for any distinct ${x_1,\dots,x_{k+1} \in A}$, at least one of the ${x_i+x_j}$, ${1 \leq i < j \leq k+1}$, must also lie in ${A}$. Roughly speaking, the arithmetic removal lemma allows one to “almost” remove the requirement that ${x_1,\dots,x_{k+1}}$ be distinct, which basically now means that ${x \in A \implies 2x \in A}$ for almost all ${x \in A}$. This near-dilation symmetry, when combined with the hypothesis that ${|G|}$ has no small prime factors, gives a lot of “dispersion” in the Fourier coefficients of ${1_A}$ which can now be exploited to prove the theorem.

The second tool is the following structure theorem, which is the main result of our paper, and goes a fair ways towards classifying sets ${A}$ for which ${\phi(A)}$ is small:

Theorem 3 Let ${A}$ be a finite subset of an arbitrary additive group ${G}$, with ${\phi(A) \leq k}$. Then one can find finite subgroups ${H_1,\dots,H_m}$ with ${m \leq k}$ such that ${|A \cap H_i| \gg_k |H_i|}$ and ${|A \backslash (H_1 \cup \dots \cup H_m)| \ll_k 1}$. Furthermore, if ${m=k}$, then the exceptional set ${A \backslash (H_1 \cup \dots \cup H_m)}$ is empty.

Roughly speaking, this theorem shows that the example of the union of ${k}$ subgroups mentioned earlier is more or less the “only” example of sets ${A}$ with ${\phi(A) \leq k}$, modulo the addition of some small exceptional sets and some refinement of the subgroups to dense subsets.

This theorem has the flavour of other inverse theorems in additive combinatorics, such as Freiman’s theorem, and indeed one can use Freiman’s theorem (and related tools, such as the Balog-Szemeredi theorem) to easily get a weaker version of this theorem. Indeed, if there are no sum-free subsets of ${A}$ of order ${k+1}$, then a fraction ${\gg_k 1}$ of all pairs ${a,b}$ in ${A}$ must have their sum also in ${A}$ (otherwise one could take ${k+1}$ random elements of ${A}$ and they would be sum-free in ${A}$ with positive probability). From this and the Balog-Szemeredi theorem and Freiman’s theorem (in arbitrary abelian groups, as established by Green and Ruzsa), we see that ${A}$ must be “commensurate” with a “coset progression” ${H+P}$ of bounded rank. One can then eliminate the torsion-free component ${P}$ of this coset progression by a number of methods (e.g. by using variants of the argument in Proposition 1), with the upshot being that one can locate a finite group ${H_1}$ that has large intersection with ${A}$.

At this point it is tempting to simply remove ${H_1}$ from ${A}$ and iterate. But one runs into a technical difficulty that removing a set such as ${H_1}$ from ${A}$ can alter the quantity ${\phi(A)}$ in unpredictable ways, so one has to still keep ${H_1}$ around when analysing the residual set ${A \backslash H_1}$. A second difficulty is that the latter set ${A \backslash H_1}$ could be considerably smaller than ${A}$ or ${H_1}$, but still large in absolute terms, so in particular any error term whose size is only bounded by ${\varepsilon |A|}$ for a small ${\varepsilon}$ could be massive compared with the residual set ${A\backslash H_1}$, and so such error terms would be unacceptable. One can get around these difficulties if one first performs some preliminary “normalisation” of the group ${H_1}$, so that the residual set ${A \backslash H_1}$ does not intersect any coset of ${H_1}$ too strongly. The arguments become even more complicated when one starts removing more than one group ${H_1,\dots,H_i}$ from ${A}$ and analyses the residual set ${A \backslash (H_1 \cup \dots \cup H_i)}$; indeed the “epsilon management” involved became so fearsomely intricate that we were forced to use a nonstandard analysis formulation of the problem in order to keep the complexity of the argument at a reasonable level (cf. my previous blog post on this topic). One drawback of doing so is that we have no effective bounds for the implied constants in our main theorem; it would be of interest to obtain a more direct proof of our main theorem that would lead to effective bounds.

I’ve just uploaded two related papers to the arXiv:

This pair of papers is an outgrowth of these two recent blog posts and the ensuing discussion. In the first paper, we establish the following logarithmically averaged version of the Chowla conjecture (in the case ${k=2}$ of two-point correlations (or “pair correlations”)):

Theorem 1 (Logarithmically averaged Chowla conjecture) Let ${a_1,a_2}$ be natural numbers, and let ${b_1,b_2}$ be integers such that ${a_1 b_2 - a_2 b_1 \neq 0}$. Let ${1 \leq \omega(x) \leq x}$ be a quantity depending on ${x}$ that goes to infinity as ${x \rightarrow \infty}$. Let ${\lambda}$ denote the Liouville function. Then one has

$\displaystyle \sum_{x/\omega(x) < n \leq x} \frac{\lambda(a_1 n + b_1) \lambda(a_2 n+b_2)}{n} = o( \log \omega(x) ) \ \ \ \ \ (1)$

as ${x \rightarrow \infty}$.

Thus for instance one has

$\displaystyle \sum_{n \leq x} \frac{\lambda(n) \lambda(n+1)}{n} = o(\log x). \ \ \ \ \ (2)$

For comparison, the non-averaged Chowla conjecture would imply that

$\displaystyle \sum_{n \leq x} \lambda(n) \lambda(n+1) = o(x) \ \ \ \ \ (3)$

which is a strictly stronger estimate than (2), and remains open.

The arguments also extend to other completely multiplicative functions than the Liouville function. In particular, one obtains a slightly averaged version of the non-asymptotic Elliott conjecture that was shown in the previous blog post to imply a positive solution to the Erdos discrepancy problem. The averaged version of the conjecture established in this paper is slightly weaker than the one assumed in the previous blog post, but it turns out that the arguments there can be modified without much difficulty to accept this averaged Elliott conjecture as input. In particular, we obtain an unconditional solution to the Erdos discrepancy problem as a consequence; this is detailed in the second paper listed above. In fact we can also handle the vector-valued version of the Erdos discrepancy problem, in which the sequence ${f(1), f(2), \dots}$ takes values in the unit sphere of an arbitrary Hilbert space, rather than in ${\{-1,+1\}}$.

Estimates such as (2) or (3) are known to be subject to the “parity problem” (discussed numerous times previously on this blog), which roughly speaking means that they cannot be proven solely using “linear” estimates on functions such as the von Mangoldt function. However, it is known that the parity problem can be circumvented using “bilinear” estimates, and this is basically what is done here.

We now describe in informal terms the proof of Theorem 1, focusing on the model case (2) for simplicity. Suppose for contradiction that the left-hand side of (2) was large and (say) positive. Using the multiplicativity ${\lambda(pn) = -\lambda(n)}$, we conclude that

$\displaystyle \sum_{n \leq x} \frac{\lambda(n) \lambda(n+p) 1_{p|n}}{n}$

is also large and positive for all primes ${p}$ that are not too large; note here how the logarithmic averaging allows us to leave the constraint ${n \leq x}$ unchanged. Summing in ${p}$, we conclude that

$\displaystyle \sum_{n \leq x} \frac{ \sum_{p \in {\mathcal P}} \lambda(n) \lambda(n+p) 1_{p|n}}{n}$

is large and positive for any given set ${{\mathcal P}}$ of medium-sized primes. By a standard averaging argument, this implies that

$\displaystyle \frac{1}{H} \sum_{j=1}^H \sum_{p \in {\mathcal P}} \lambda(n+j) \lambda(n+p+j) 1_{p|n+j} \ \ \ \ \ (4)$

is large for many choices of ${n}$, where ${H}$ is a medium-sized parameter at our disposal to choose, and we take ${{\mathcal P}}$ to be some set of primes that are somewhat smaller than ${H}$. (A similar approach was taken in this recent paper of Matomaki, Radziwill, and myself to study sign patterns of the Möbius function.) To obtain the required contradiction, one thus wants to demonstrate significant cancellation in the expression (4). As in that paper, we view ${n}$ as a random variable, in which case (4) is essentially a bilinear sum of the random sequence ${(\lambda(n+1),\dots,\lambda(n+H))}$ along a random graph ${G_{n,H}}$ on ${\{1,\dots,H\}}$, in which two vertices ${j, j+p}$ are connected if they differ by a prime ${p}$ in ${{\mathcal P}}$ that divides ${n+j}$. A key difficulty in controlling this sum is that for randomly chosen ${n}$, the sequence ${(\lambda(n+1),\dots,\lambda(n+H))}$ and the graph ${G_{n,H}}$ need not be independent. To get around this obstacle we introduce a new argument which we call the “entropy decrement argument” (in analogy with the “density increment argument” and “energy increment argument” that appear in the literature surrounding Szemerédi’s theorem on arithmetic progressions, and also reminiscent of the “entropy compression argument” of Moser and Tardos, discussed in this previous post). This argument, which is a simple consequence of the Shannon entropy inequalities, can be viewed as a quantitative version of the standard subadditivity argument that establishes the existence of Kolmogorov-Sinai entropy in topological dynamical systems; it allows one to select a scale parameter ${H}$ (in some suitable range ${[H_-,H_+]}$) for which the sequence ${(\lambda(n+1),\dots,\lambda(n+H))}$ and the graph ${G_{n,H}}$ exhibit some weak independence properties (or more precisely, the mutual information between the two random variables is small).

Informally, the entropy decrement argument goes like this: if the sequence ${(\lambda(n+1),\dots,\lambda(n+H))}$ has significant mutual information with ${G_{n,H}}$, then the entropy of the sequence ${(\lambda(n+1),\dots,\lambda(n+H'))}$ for ${H' > H}$ will grow a little slower than linearly, due to the fact that the graph ${G_{n,H}}$ has zero entropy (knowledge of ${G_{n,H}}$ more or less completely determines the shifts ${G_{n+kH,H}}$ of the graph); this can be formalised using the classical Shannon inequalities for entropy (and specifically, the non-negativity of conditional mutual information). But the entropy cannot drop below zero, so by increasing ${H}$ as necessary, at some point one must reach a metastable region (cf. the finite convergence principle discussed in this previous blog post), within which very little mutual information can be shared between the sequence ${(\lambda(n+1),\dots,\lambda(n+H))}$ and the graph ${G_{n,H}}$. Curiously, for the application it is not enough to have a purely quantitative version of this argument; one needs a quantitative bound (which gains a factor of a bit more than ${\log H}$ on the trivial bound for mutual information), and this is surprisingly delicate (it ultimately comes down to the fact that the series ${\sum_{j \geq 2} \frac{1}{j \log j \log\log j}}$ diverges, which is only barely true).

Once one locates a scale ${H}$ with the low mutual information property, one can use standard concentration of measure results such as the Hoeffding inequality to approximate (4) by the significantly simpler expression

$\displaystyle \frac{1}{H} \sum_{j=1}^H \sum_{p \in {\mathcal P}} \frac{\lambda(n+j) \lambda(n+p+j)}{p}. \ \ \ \ \ (5)$

The important thing here is that Hoeffding’s inequality gives exponentially strong bounds on the failure probability, which is needed to counteract the logarithms that are inevitably present whenever trying to use entropy inequalities. The expression (5) can then be controlled in turn by an application of the Hardy-Littlewood circle method and a non-trivial estimate

$\displaystyle \sup_\alpha \frac{1}{X} \int_X^{2X} |\frac{1}{H} \sum_{x \leq n \leq x+H} \lambda(n) e(\alpha n)|\ dx = o(1) \ \ \ \ \ (6)$

for averaged short sums of a modulated Liouville function established in another recent paper by Matomäki, Radziwill and myself.

When one uses this method to study more general sums such as

$\displaystyle \sum_{n \leq x} \frac{g_1(n) g_2(n+1)}{n},$

one ends up having to consider expressions such as

$\displaystyle \frac{1}{H} \sum_{j=1}^H \sum_{p \in {\mathcal P}} c_p \frac{g_1(n+j) g_2(n+p+j)}{p}.$

where ${c_p}$ is the coefficient ${c_p := \overline{g_1}(p) \overline{g_2}(p)}$. When attacking this sum with the circle method, one soon finds oneself in the situation of wanting to locate the large Fourier coefficients of the exponential sum

$\displaystyle S(\alpha) := \sum_{p \in {\mathcal P}} \frac{c_p}{p} e^{2\pi i \alpha p}.$

In many cases (such as in the application to the Erdös discrepancy problem), the coefficient ${c_p}$ is identically ${1}$, and one can understand this sum satisfactorily using the classical results of Vinogradov: basically, ${S(\alpha)}$ is large when ${\alpha}$ lies in a “major arc” and is small when it lies in a “minor arc”. For more general functions ${g_1,g_2}$, the coefficients ${c_p}$ are more or less arbitrary; the large values of ${S(\alpha)}$ are no longer confined to the major arc case. Fortunately, even in this general situation one can use a restriction theorem for the primes established some time ago by Ben Green and myself to show that there are still only a bounded number of possible locations ${\alpha}$ (up to the uncertainty mandated by the Heisenberg uncertainty principle) where ${S(\alpha)}$ is large, and we can still conclude by using (6). (Actually, as recently pointed out to me by Ben, one does not need the full strength of our result; one only needs the ${L^4}$ restriction theorem for the primes, which can be proven fairly directly using Plancherel’s theorem and some sieve theory.)

It is tempting to also use the method to attack higher order cases of the (logarithmically) averaged Chowla conjecture, for instance one could try to prove the estimate

$\displaystyle \sum_{n \leq x} \frac{\lambda(n) \lambda(n+1) \lambda(n+2)}{n} = o(\log x).$

The above arguments reduce matters to obtaining some non-trivial cancellation for sums of the form

$\displaystyle \frac{1}{H} \sum_{j=1}^H \sum_{p \in {\mathcal P}} \frac{\lambda(n+j) \lambda(n+p+j) \lambda(n+2p+j)}{p}.$

A little bit of “higher order Fourier analysis” (as was done for very similar sums in the ergodic theory context by Frantzikinakis-Host-Kra and Wooley-Ziegler) lets one control this sort of sum if one can establish a bound of the form

$\displaystyle \frac{1}{X} \int_X^{2X} \sup_\alpha |\frac{1}{H} \sum_{x \leq n \leq x+H} \lambda(n) e(\alpha n)|\ dx = o(1) \ \ \ \ \ (7)$

where ${X}$ goes to infinity and ${H}$ is a very slowly growing function of ${X}$. This looks very similar to (6), but the fact that the supremum is now inside the integral makes the problem much more difficult. However it looks worth attacking (7) further, as this estimate looks like it should have many nice applications (beyond just the ${k=3}$ case of the logarithmically averaged Chowla or Elliott conjectures, which is already interesting).

For higher ${k}$ than ${k=3}$, the same line of analysis requires one to replace the linear phase ${e(\alpha n)}$ by more complicated phases, such as quadratic phases ${e(\alpha n^2 + \beta n)}$ or even ${k-2}$-step nilsequences. Given that (7) is already beyond the reach of current literature, these even more complicated expressions are also unavailable at present, but one can imagine that they will eventually become tractable, in which case we would obtain an averaged form of the Chowla conjecture for all ${k}$, which would have a number of consequences (such as a logarithmically averaged version of Sarnak’s conjecture, as per this blog post).

It would of course be very nice to remove the logarithmic averaging, and be able to establish bounds such as (3). I did attempt to do so, but I do not see a way to use the entropy decrement argument in a manner that does not require some sort of averaging of logarithmic type, as it requires one to pick a scale ${H}$ that one cannot specify in advance, which is not a problem for logarithmic averages (which are quite stable with respect to dilations) but is problematic for ordinary averages. But perhaps the problem can be circumvented by some clever modification of the argument. One possible approach would be to start exploiting multiplicativity at products of primes, and not just individual primes, to try to keep the scale fixed, but this makes the concentration of measure part of the argument much more complicated as one loses some independence properties (coming from the Chinese remainder theorem) which allowed one to conclude just from the Hoeffding inequality.

This week I have been at a Banff workshop “Combinatorics meets Ergodic theory“, focused on the combinatorics surrounding Szemerédi’s theorem and the Gowers uniformity norms on one hand, and the ergodic theory surrounding Furstenberg’s multiple recurrence theorem and the Host-Kra structure theory on the other. This was quite a fruitful workshop, and directly inspired the various posts this week on this blog. Incidentally, BIRS being as efficient as it is, videos for this week’s talks are already online.

As mentioned in the previous two posts, Ben Green, Tamar Ziegler, and myself proved the following inverse theorem for the Gowers norms:

Theorem 1 (Inverse theorem for Gowers norms) Let ${N \geq 1}$ and ${s \geq 1}$ be integers, and let ${\delta > 0}$. Suppose that ${f: {\bf Z} \rightarrow [-1,1]}$ is a function supported on ${[N] := \{1,\dots,N\}}$ such that

$\displaystyle \frac{1}{N^{s+2}} \sum_{n,h_1,\dots,h_{s+1} \in {\bf Z}} \prod_{\omega \in \{0,1\}^{s+1}} f(n+\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}) \geq \delta.$

Then there exists a filtered nilmanifold ${G/\Gamma}$ of degree ${\leq s}$ and complexity ${O_{s,\delta}(1)}$, a polynomial sequence ${g: {\bf Z} \rightarrow G}$, and a Lipschitz function ${F: G/\Gamma \rightarrow {\bf R}}$ of Lipschitz constant ${O_{s,\delta}(1)}$ such that

$\displaystyle \frac{1}{N} \sum_n f(n) F(g(n) \Gamma) \gg_{s,\delta} 1.$

There is a higher dimensional generalisation, which first appeared explicitly (in a more general form) in this preprint of Szegedy (which used a slightly different argument than the one of Ben, Tammy, and myself; see also this previous preprint of Szegedy with related results):

Theorem 2 (Inverse theorem for multidimensional Gowers norms) Let ${N \geq 1}$ and ${s,d \geq 1}$ be integers, and let ${\delta > 0}$. Suppose that ${f: {\bf Z}^d \rightarrow [-1,1]}$ is a function supported on ${[N]^d}$ such that

$\displaystyle \frac{1}{N^{d(s+2)}} \sum_{n,h_1,\dots,h_{s+1} \in {\bf Z}^d} \prod_{\omega \in \{0,1\}^{s+1}} f(n+\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}) \geq \delta. \ \ \ \ \ (1)$

Then there exists a filtered nilmanifold ${G/\Gamma}$ of degree ${\leq s}$ and complexity ${O_{s,\delta,d}(1)}$, a polynomial sequence ${g: {\bf Z}^d \rightarrow G}$, and a Lipschitz function ${F: G/\Gamma \rightarrow {\bf R}}$ of Lipschitz constant ${O_{s,\delta,d}(1)}$ such that

$\displaystyle \frac{1}{N^d} \sum_{n \in {\bf Z}^d} f(n) F(g(n) \Gamma) \gg_{s,\delta,d} 1.$

The ${d=2}$ case of this theorem was recently used by Wenbo Sun. One can replace the polynomial sequence with a linear sequence if desired by using a lifting trick (essentially due to Furstenberg, but which appears explicitly in Appendix C of my paper with Ben and Tammy).

In this post I would like to record a very neat and simple observation of Ben Green and Nikos Frantzikinakis, that uses the tool of Freiman isomorphisms to derive Theorem 2 as a corollary of the one-dimensional theorem. Namely, consider the linear map ${\phi: {\bf Z}^d \rightarrow {\bf Z}}$ defined by

$\displaystyle \phi( n_1,\dots,n_d ) := \sum_{i=1}^d (10 N)^{i-1} n_i,$

that is to say ${\phi}$ is the digit string base ${10N}$ that has digits ${n_d \dots n_1}$. This map is a linear map from ${[N]^d}$ to a subset of ${[d 10^d N^d]}$ of density ${1/(d10^d)}$. Furthermore it has the following “Freiman isomorphism” property: if ${n, h_1,\dots,h_{s+1}}$ lie in ${{\bf Z}}$ with ${n + \omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}}$ in the image set ${\phi( [N]^d )}$ of ${[N]^d}$ for all ${\omega}$, then there exist (unique) lifts ${\tilde n \in {\bf Z}^d, \tilde h_1,\dots,\tilde h_{s+1} \in {\bf Z}}$ such that

$\displaystyle \tilde n + \omega_1 \tilde h_1 + \dots + \omega_{s+1} \tilde h_{s+1} \in [N]^d$

and

$\displaystyle \phi( \tilde n + \omega_1 \tilde h_1 + \dots + \omega_{s+1} \tilde h_{s+1} ) = n + \omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}$

for all ${\omega}$. Indeed, the injectivity of ${\phi}$ on ${[N]^d}$ uniquely determines the sum ${\tilde n + \omega_1 \tilde h_1 + \dots + \omega_{s+1} \tilde h_{s+1}}$ for each ${\omega}$, and one can use base ${10N}$ arithmetic to verify that the alternating sum of these sums on any ${2}$-facet of the cube ${\{0,1\}^{s+1}}$ vanishes, which gives the claim. (In the language of additive combinatorics, the point is that ${\phi}$ is a Freiman isomorphism of order (say) ${8}$ on ${[N]^d}$.)

Now let ${\tilde f: {\bf Z} \rightarrow [-1,1]}$ be the function defined by setting ${\tilde f( \phi(n) ) := f(n)}$ whenever ${n \in [N]^d}$, with ${\tilde f}$ vanishing outside of ${\phi([N]^d)}$. If ${f}$ obeys (1), then from the above Freiman isomorphism property we have

$\displaystyle \frac{1}{N^{d(s+2)}} \sum_{n, h_1,\dots,h_{s+1} \in {\bf Z}} \prod_{\omega \in \{0,1\}^{s+1}} \tilde f(n+\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}) \geq \delta.$

Applying the one-dimensional inverse theorem (Theorem 1), with ${\delta}$ reduced by a factor of ${d 10^d}$ and ${N}$ replaced by ${d 10^d N^d}$, this implies the existence of a filtered nilmanifold ${G/\Gamma}$ of degree ${\leq s}$ and complexity ${O_{s,\delta,d}(1)}$, a polynomial sequence ${g: {\bf Z} \rightarrow G}$, and a Lipschitz function ${F: G/\Gamma \rightarrow {\bf R}}$ of Lipschitz constant ${O_{s,\delta,d}(1)}$ such that

$\displaystyle \frac{1}{N^{d(s+2)}} \sum_{n \in {\bf Z}} \tilde f(n) F(g(n) \Gamma) \gg_{s,\delta,d} 1$

which by the Freiman isomorphism property again implies that

$\displaystyle \frac{1}{N^{d(s+2)}} \sum_{n \in {\bf Z}^d} f(n) F(g(\phi(n)) \Gamma) \gg_{s,\delta,d} 1.$

But the map ${n \mapsto g(\phi(n))}$ is clearly a polynomial map from ${{\bf Z}^d}$ to ${G}$ (the composition of two polynomial maps is polynomial, see e.g. Appendix B of my paper with Ben and Tammy), and the claim follows.

Remark 3 This trick appears to be largely restricted to the case of boundedly generated groups such as ${{\bf Z}^d}$; I do not see any easy way to deduce an inverse theorem for, say, ${\bigcup_{n=1}^\infty {\mathbb F}_p^n}$ from the ${{\bf Z}}$-inverse theorem by this method.

Remark 4 By combining this argument with the one in the previous post, one can obtain a weak ergodic inverse theorem for ${{\bf Z}^d}$-actions. Interestingly, the Freiman isomorphism argument appears to be difficult to implement directly in the ergodic category; in particular, there does not appear to be an obvious direct way to derive the Host-Kra inverse theorem for ${{\bf Z}^d}$ actions (a result first obtained in the PhD thesis of Griesmer) from the counterpart for ${{\bf Z}}$ actions.

Note: this post is of a particularly technical nature, in particular presuming familiarity with nilsequences, nilsystems, characteristic factors, etc., and is primarily intended for experts.

As mentioned in the previous post, Ben Green, Tamar Ziegler, and myself proved the following inverse theorem for the Gowers norms:

Theorem 1 (Inverse theorem for Gowers norms) Let ${N \geq 1}$ and ${s \geq 1}$ be integers, and let ${\delta > 0}$. Suppose that ${f: {\bf Z} \rightarrow [-1,1]}$ is a function supported on ${[N] := \{1,\dots,N\}}$ such that

$\displaystyle \frac{1}{N^{s+2}} \sum_{n,h_1,\dots,h_{s+1}} \prod_{\omega \in \{0,1\}^{s+1}} f(n+\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}) \geq \delta.$

Then there exists a filtered nilmanifold ${G/\Gamma}$ of degree ${\leq s}$ and complexity ${O_{s,\delta}(1)}$, a polynomial sequence ${g: {\bf Z} \rightarrow G}$, and a Lipschitz function ${F: G/\Gamma \rightarrow {\bf R}}$ of Lipschitz constant ${O_{s,\delta}(1)}$ such that

$\displaystyle \frac{1}{N} \sum_n f(n) F(g(n) \Gamma) \gg_{s,\delta} 1.$

This result was conjectured earlier by Ben Green and myself; this conjecture was strongly motivated by an analogous inverse theorem in ergodic theory by Host and Kra, which we formulate here in a form designed to resemble Theorem 1 as closely as possible:

Theorem 2 (Inverse theorem for Gowers-Host-Kra seminorms) Let ${s \geq 1}$ be an integer, and let ${(X, T)}$ be an ergodic, countably generated measure-preserving system. Suppose that one has

$\displaystyle \lim_{N \rightarrow \infty} \frac{1}{N^{s+1}} \sum_{h_1,\dots,h_{s+1} \in [N]} \int_X \prod_{\omega \in \{0,1\}^{s+1}} f(T^{\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}}x)\ d\mu(x)$

$\displaystyle > 0$

for all non-zero ${f \in L^\infty(X)}$ (all ${L^p}$ spaces are real-valued in this post). Then ${(X,T)}$ is an inverse limit (in the category of measure-preserving systems, up to almost everywhere equivalence) of ergodic degree ${\leq s}$ nilsystems, that is to say systems of the form ${(G/\Gamma, x \mapsto gx)}$ for some degree ${\leq s}$ filtered nilmanifold ${G/\Gamma}$ and a group element ${g \in G}$ that acts ergodically on ${G/\Gamma}$.

It is a natural question to ask if there is any logical relationship between the two theorems. In the finite field category, one can deduce the combinatorial inverse theorem from the ergodic inverse theorem by a variant of the Furstenberg correspondence principle, as worked out by Tamar Ziegler and myself, however in the current context of ${{\bf Z}}$-actions, the connection is less clear.

One can split Theorem 2 into two components:

Theorem 3 (Weak inverse theorem for Gowers-Host-Kra seminorms) Let ${s \geq 1}$ be an integer, and let ${(X, T)}$ be an ergodic, countably generated measure-preserving system. Suppose that one has

$\displaystyle \lim_{N \rightarrow \infty} \frac{1}{N^{s+1}} \sum_{h_1,\dots,h_{s+1} \in [N]} \int_X \prod_{\omega \in \{0,1\}^{s+1}} T^{\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}} f\ d\mu$

$\displaystyle > 0$

for all non-zero ${f \in L^\infty(X)}$, where ${T^h f := f \circ T^h}$. Then ${(X,T)}$ is a factor of an inverse limit of ergodic degree ${\leq s}$ nilsystems.

Theorem 4 (Pro-nilsystems closed under factors) Let ${s \geq 1}$ be an integer. Then any factor of an inverse limit of ergodic degree ${\leq s}$ nilsystems, is again an inverse limit of ergodic degree ${\leq s}$ nilsystems.

Indeed, it is clear that Theorem 2 implies both Theorem 3 and Theorem 4, and conversely that the two latter theorems jointly imply the former. Theorem 4 is, in principle, purely a fact about nilsystems, and should have an independent proof, but this is not known; the only known proofs go through the full machinery needed to prove Theorem 2 (or the closely related theorem of Ziegler). (However, the fact that a factor of a nilsystem is again a nilsystem was established previously by Parry.)

The purpose of this post is to record a partial implication in reverse direction to the correspondence principle:

Proposition 5 Theorem 1 implies Theorem 3.

As mentioned at the start of the post, a fair amount of familiarity with the area is presumed here, and some routine steps will be presented with only a fairly brief explanation.

A few years ago, Ben Green, Tamar Ziegler, and myself proved the following (rather technical-looking) inverse theorem for the Gowers norms:

Theorem 1 (Discrete inverse theorem for Gowers norms) Let ${N \geq 1}$ and ${s \geq 1}$ be integers, and let ${\delta > 0}$. Suppose that ${f: {\bf Z} \rightarrow [-1,1]}$ is a function supported on ${[N] := \{1,\dots,N\}}$ such that

$\displaystyle \frac{1}{N^{s+2}} \sum_{n,h_1,\dots,h_{s+1}} \prod_{\omega \in \{0,1\}^{s+1}} f(n+\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1}) \geq \delta.$

Then there exists a filtered nilmanifold ${G/\Gamma}$ of degree ${\leq s}$ and complexity ${O_{s,\delta}(1)}$, a polynomial sequence ${g: {\bf Z} \rightarrow G}$, and a Lipschitz function ${F: G/\Gamma \rightarrow {\bf R}}$ of Lipschitz constant ${O_{s,\delta}(1)}$ such that

$\displaystyle \frac{1}{N} \sum_n f(n) F(g(n) \Gamma) \gg_{s,\delta} 1.$

For the definitions of “filtered nilmanifold”, “degree”, “complexity”, and “polynomial sequence”, see the paper of Ben, Tammy, and myself. (I should caution the reader that this blog post will presume a fair amount of familiarity with this subfield of additive combinatorics.) This result has a number of applications, for instance to establishing asymptotics for linear equations in the primes, but this will not be the focus of discussion here.

The purpose of this post is to record the observation that this “discrete” inverse theorem, together with an equidistribution theorem for nilsequences that Ben and I worked out in a separate paper, implies a continuous version:

Theorem 2 (Continuous inverse theorem for Gowers norms) Let ${s \geq 1}$ be an integer, and let ${\delta>0}$. Suppose that ${f: {\bf R} \rightarrow [-1,1]}$ is a measurable function supported on ${[0,1]}$ such that

$\displaystyle \int_{{\bf R}^{s+1}} \prod_{\omega \in \{0,1\}^{s+1}} f(t+\omega_1 h_1 + \dots + \omega_{s+1} h_{s+1})\ dt dh_1 \dots dh_{s+1} \geq \delta. \ \ \ \ \ (1)$

Then there exists a filtered nilmanifold ${G/\Gamma}$ of degree ${\leq s}$ and complexity ${O_{s,\delta}(1)}$, a (smooth) polynomial sequence ${g: {\bf R} \rightarrow G}$, and a Lipschitz function ${F: G/\Gamma \rightarrow {\bf R}}$ of Lipschitz constant ${O_{s,\delta}(1)}$ such that

$\displaystyle \int_{\bf R} f(t) F(g(t) \Gamma)\ dt \gg_{s,\delta} 1.$

The interval ${[0,1]}$ can be easily replaced with any other fixed interval by a change of variables. A key point here is that the bounds are completely uniform in the choice of ${f}$. Note though that the coefficients of ${g}$ can be arbitrarily large (and this is necessary, as can be seen just by considering functions of the form ${f(t) = \cos( \xi t)}$ for some arbitrarily large frequency ${\xi}$).

It is likely that one could prove Theorem 2 by carefully going through the proof of Theorem 1 and replacing all instances of ${{\bf Z}}$ with ${{\bf R}}$ (and making appropriate modifications to the argument to accommodate this). However, the proof of Theorem 1 is quite lengthy. Here, we shall proceed by the usual limiting process of viewing the continuous interval ${[0,1]}$ as a limit of the discrete interval ${\frac{1}{N} \cdot [N]}$ as ${N \rightarrow \infty}$. However there will be some problems taking the limit due to a failure of compactness, and specifically with regards to the coefficients of the polynomial sequence ${g: {\bf N} \rightarrow G}$ produced by Theorem 1, after normalising these coefficients by ${N}$. Fortunately, a factorisation theorem from a paper of Ben Green and myself resolves this problem by splitting ${g}$ into a “smooth” part which does enjoy good compactness properties, as well as “totally equidistributed” and “periodic” parts which can be eliminated using the measurability (and thus, approximate smoothness), of ${f}$.

Szemerédi’s theorem asserts that any subset of the integers of positive upper density contains arbitrarily large arithmetic progressions. Here is an equivalent quantitative form of this theorem:

Theorem 1 (Szemerédi’s theorem) Let ${N}$ be a positive integer, and let ${f: {\bf Z}/N{\bf Z} \rightarrow [0,1]}$ be a function with ${{\bf E}_{x \in {\bf Z}/N{\bf Z}} f(x) \geq \delta}$ for some ${\delta>0}$, where we use the averaging notation ${{\bf E}_{x \in A} f(x) := \frac{1}{|A|} \sum_{x \in A} f(x)}$, ${{\bf E}_{x,r \in A} f(x) := \frac{1}{|A|^2} \sum_{x, r \in A} f(x)}$, etc.. Then for ${k \geq 3}$ we have

$\displaystyle {\bf E}_{x,r \in {\bf Z}/N{\bf Z}} f(x) f(x+r) \dots f(x+(k-1)r) \geq c(k,\delta)$

for some ${c(k,\delta)>0}$ depending only on ${k,\delta}$.

The equivalence is basically thanks to an averaging argument of Varnavides; see for instance Chapter 11 of my book with Van Vu or this previous blog post for a discussion. We have removed the cases ${k=1,2}$ as they are trivial and somewhat degenerate.

There are now many proofs of this theorem. Some time ago, I took an ergodic-theoretic proof of Furstenberg and converted it to a purely finitary proof of the theorem. The argument used some simplifying innovations that had been developed since the original work of Furstenberg (in particular, deployment of the Gowers uniformity norms, as well as a “dual” norm that I called the uniformly almost periodic norm, and an emphasis on van der Waerden’s theorem for handling the “compact extension” component of the argument). But the proof was still quite messy. However, as discussed in this previous blog post, messy finitary proofs can often be cleaned up using nonstandard analysis. Thus, there should be a nonstandard version of the Furstenberg ergodic theory argument that is relatively clean. I decided (after some encouragement from Ben Green and Isaac Goldbring) to write down most of the details of this argument in this blog post, though for sake of brevity I will skim rather quickly over arguments that were already discussed at length in other blog posts. In particular, I will presume familiarity with nonstandard analysis (in particular, the notion of a standard part of a bounded real number, and the Loeb measure construction), see for instance this previous blog post for a discussion.

In analytic number theory, there is a well known analogy between the prime factorisation of a large integer, and the cycle decomposition of a large permutation; this analogy is central to the topic of “anatomy of the integers”, as discussed for instance in this survey article of Granville. Consider for instance the following two parallel lists of facts (stated somewhat informally). Firstly, some facts about the prime factorisation of large integers:

• Every positive integer ${m}$ has a prime factorisation

$\displaystyle m = p_1 p_2 \dots p_r$

into (not necessarily distinct) primes ${p_1,\dots,p_r}$, which is unique up to rearrangement. Taking logarithms, we obtain a partition

$\displaystyle \log m = \log p_1 + \log p_2 + \dots + \log p_r$

of ${\log m}$.

• (Prime number theorem) A randomly selected integer ${m}$ of size ${m \sim N}$ will be prime with probability ${\approx \frac{1}{\log N}}$ when ${N}$ is large.
• If ${m \sim N}$ is a randomly selected large integer of size ${N}$, and ${p = p_i}$ is a randomly selected prime factor of ${m = p_1 \dots p_r}$ (with each index ${i}$ being chosen with probability ${\frac{\log p_i}{\log m}}$), then ${\log p_i}$ is approximately uniformly distributed between ${0}$ and ${\log N}$. (See Proposition 9 of this previous blog post.)
• The set of real numbers ${\{ \frac{\log p_i}{\log m}: i=1,\dots,r \}}$ arising from the prime factorisation ${m = p_1 \dots p_r}$ of a large random number ${m \sim N}$ converges (away from the origin, and in a suitable weak sense) to the Poisson-Dirichlet process in the limit ${N \rightarrow \infty}$. (See the previously mentioned blog post for a definition of the Poisson-Dirichlet process, and a proof of this claim.)

Now for the facts about the cycle decomposition of large permutations:

• Every permutation ${\sigma \in S_n}$ has a cycle decomposition

$\displaystyle \sigma = C_1 \dots C_r$

into disjoint cycles ${C_1,\dots,C_r}$, which is unique up to rearrangement, and where we count each fixed point of ${\sigma}$ as a cycle of length ${1}$. If ${|C_i|}$ is the length of the cycle ${C_i}$, we obtain a partition

$\displaystyle n = |C_1| + \dots + |C_r|$

of ${n}$.

• (Prime number theorem for permutations) A randomly selected permutation of ${S_n}$ will be an ${n}$-cycle with probability exactly ${1/n}$. (This was noted in this previous blog post.)
• If ${\sigma}$ is a random permutation in ${S_n}$, and ${C_i}$ is a randomly selected cycle of ${\sigma}$ (with each ${i}$ being selected with probability ${|C_i|/n}$), then ${|C_i|}$ is exactly uniformly distributed on ${\{1,\dots,n\}}$. (See Proposition 8 of this blog post.)
• The set of real numbers ${\{ \frac{|C_i|}{n} \}}$ arising from the cycle decomposition ${\sigma = C_1 \dots C_r}$ of a random permutation ${\sigma \in S_n}$ converges (in a suitable sense) to the Poisson-Dirichlet process in the limit ${n \rightarrow \infty}$. (Again, see this previous blog post for details.)

See this previous blog post (or the aforementioned article of Granville, or the Notices article of Arratia, Barbour, and Tavaré) for further exploration of the analogy between prime factorisation of integers and cycle decomposition of permutations.

There is however something unsatisfying about the analogy, in that it is not clear why there should be such a kinship between integer prime factorisation and permutation cycle decomposition. It turns out that the situation is clarified if one uses another fundamental analogy in number theory, namely the analogy between integers and polynomials ${P \in {\mathbf F}_q[T]}$ over a finite field ${{\mathbf F}_q}$, discussed for instance in this previous post; this is the simplest case of the more general function field analogy between number fields and function fields. Just as we restrict attention to positive integers when talking about prime factorisation, it will be reasonable to restrict attention to monic polynomials ${P}$. We then have another analogous list of facts, proven very similarly to the corresponding list of facts for the integers:

• Every monic polynomial ${f \in {\mathbf F}_q[T]}$ has a factorisation

$\displaystyle f = P_1 \dots P_r$

into irreducible monic polynomials ${P_1,\dots,P_r \in {\mathbf F}_q[T]}$, which is unique up to rearrangement. Taking degrees, we obtain a partition

$\displaystyle \hbox{deg} f = \hbox{deg} P_1 + \dots + \hbox{deg} P_r$

of ${\hbox{deg} f}$.

• (Prime number theorem for polynomials) A randomly selected monic polynomial ${f \in {\mathbf F}_q[T]}$ of degree ${n}$ will be irreducible with probability ${\approx \frac{1}{n}}$ when ${q}$ is fixed and ${n}$ is large.
• If ${f \in {\mathbf F}_q[T]}$ is a random monic polynomial of degree ${n}$, and ${P_i}$ is a random irreducible factor of ${f = P_1 \dots P_r}$ (with each ${i}$ selected with probability ${\hbox{deg} P_i / n}$), then ${\hbox{deg} P_i}$ is approximately uniformly distributed in ${\{1,\dots,n\}}$ when ${q}$ is fixed and ${n}$ is large.
• The set of real numbers ${\{ \hbox{deg} P_i / n \}}$ arising from the factorisation ${f = P_1 \dots P_r}$ of a randomly selected polynomial ${f \in {\mathbf F}_q[T]}$ of degree ${n}$ converges (in a suitable sense) to the Poisson-Dirichlet process when ${q}$ is fixed and ${n}$ is large.

The above list of facts addressed the large ${n}$ limit of the polynomial ring ${{\mathbf F}_q[T]}$, where the order ${q}$ of the field is held fixed, but the degrees of the polynomials go to infinity. This is the limit that is most closely analogous to the integers ${{\bf Z}}$. However, there is another interesting asymptotic limit of polynomial rings to consider, namely the large ${q}$ limit where it is now the degree ${n}$ that is held fixed, but the order ${q}$ of the field goes to infinity. Actually to simplify the exposition we will use the slightly more restrictive limit where the characteristic ${p}$ of the field goes to infinity (again keeping the degree ${n}$ fixed), although all of the results proven below for the large ${p}$ limit turn out to be true as well in the large ${q}$ limit.

The large ${q}$ (or large ${p}$) limit is technically a different limit than the large ${n}$ limit, but in practice the asymptotic statistics of the two limits often agree quite closely. For instance, here is the prime number theorem in the large ${q}$ limit:

Theorem 1 (Prime number theorem) The probability that a random monic polynomial ${f \in {\mathbf F}_q[T]}$ of degree ${n}$ is irreducible is ${\frac{1}{n}+o(1)}$ in the limit where ${n}$ is fixed and the characteristic ${p}$ goes to infinity.

Proof: There are ${q^n}$ monic polynomials ${f \in {\mathbf F}_q[T]}$ of degree ${n}$. If ${f}$ is irreducible, then the ${n}$ zeroes of ${f}$ are distinct and lie in the finite field ${{\mathbf F}_{q^n}}$, but do not lie in any proper subfield of that field. Conversely, every element ${\alpha}$ of ${{\mathbf F}_{q^n}}$ that does not lie in a proper subfield is the root of a unique monic polynomial in ${{\mathbf F}_q[T]}$ of degree ${f}$ (the minimal polynomial of ${\alpha}$). Since the union of all the proper subfields of ${{\mathbf F}_{q^n}}$ has size ${o(q^n)}$, the total number of irreducible polynomials of degree ${n}$ is thus ${\frac{q^n - o(q^n)}{n}}$, and the claim follows. $\Box$

Remark 2 The above argument and inclusion-exclusion in fact gives the well known exact formula ${\frac{1}{n} \sum_{d|n} \mu(\frac{n}{d}) q^d}$ for the number of irreducible monic polynomials of degree ${n}$.

Now we can give a precise connection between the cycle distribution of a random permutation, and (the large ${p}$ limit of) the irreducible factorisation of a polynomial, giving a (somewhat indirect, but still connected) link between permutation cycle decomposition and integer factorisation:

Theorem 3 The partition ${\{ \hbox{deg}(P_1), \dots, \hbox{deg}(P_r) \}}$ of a random monic polynomial ${f= P_1 \dots P_r\in {\mathbf F}_q[T]}$ of degree ${n}$ converges in distribution to the partition ${\{ |C_1|, \dots, |C_r|\}}$ of a random permutation ${\sigma = C_1 \dots C_r \in S_n}$ of length ${n}$, in the limit where ${n}$ is fixed and the characteristic ${p}$ goes to infinity.

We can quickly prove this theorem as follows. We first need a basic fact:

Lemma 4 (Most polynomials square-free in large ${q}$ limit) A random monic polynomial ${f \in {\mathbf F}_q[T]}$ of degree ${n}$ will be square-free with probability ${1-o(1)}$ when ${n}$ is fixed and ${q}$ (or ${p}$) goes to infinity. In a similar spirit, two randomly selected monic polynomials ${f,g}$ of degree ${n,m}$ will be coprime with probability ${1-o(1)}$ if ${n,m}$ are fixed and ${q}$ or ${p}$ goes to infinity.

Proof: For any polynomial ${g}$ of degree ${m}$, the probability that ${f}$ is divisible by ${g^2}$ is at most ${1/q^{2m}}$. Summing over all polynomials of degree ${1 \leq m \leq n/2}$, and using the union bound, we see that the probability that ${f}$ is not squarefree is at most ${\sum_{1 \leq m \leq n/2} \frac{q^m}{q^{2m}} = o(1)}$, giving the first claim. For the second, observe from the first claim (and the fact that ${fg}$ has only a bounded number of factors) that ${fg}$ is squarefree with probability ${1-o(1)}$, giving the claim. $\Box$

Now we can prove the theorem. Elementary combinatorics tells us that the probability of a random permutation ${\sigma \in S_n}$ consisting of ${c_k}$ cycles of length ${k}$ for ${k=1,\dots,r}$, where ${c_k}$ are nonnegative integers with ${\sum_{k=1}^r k c_k = n}$, is precisely

$\displaystyle \frac{1}{\prod_{k=1}^r c_k! k^{c_k}},$

since there are ${\prod_{k=1}^r c_k! k^{c_k}}$ ways to write a given tuple of cycles ${C_1,\dots,C_r}$ in cycle notation in nondecreasing order of length, and ${n!}$ ways to select the labels for the cycle notation. On the other hand, by Theorem 1 (and using Lemma 4 to isolate the small number of cases involving repeated factors) the number of monic polynomials of degree ${n}$ that are the product of ${c_k}$ irreducible polynomials of degree ${k}$ is

$\displaystyle \frac{1}{\prod_{k=1}^r c_k!} \prod_{k=1}^r ( (\frac{1}{k}+o(1)) q^k )^{c_k} + o( q^n )$

which simplifies to

$\displaystyle \frac{1+o(1)}{\prod_{k=1}^r c_k! k^{c_k}} q^n,$

and the claim follows.

This was a fairly short calculation, but it still doesn’t quite explain why there is such a link between the cycle decomposition ${\sigma = C_1 \dots C_r}$ of permutations and the factorisation ${f = P_1 \dots P_r}$ of a polynomial. One immediate thought might be to try to link the multiplication structure of permutations in ${S_n}$ with the multiplication structure of polynomials; however, these structures are too dissimilar to set up a convincing analogy. For instance, the multiplication law on polynomials is abelian and non-invertible, whilst the multiplication law on ${S_n}$ is (extremely) non-abelian but invertible. Also, the multiplication of a degree ${n}$ and a degree ${m}$ polynomial is a degree ${n+m}$ polynomial, whereas the group multiplication law on permutations does not take a permutation in ${S_n}$ and a permutation in ${S_m}$ and return a permutation in ${S_{n+m}}$.

I recently found (after some discussions with Ben Green) what I feel to be a satisfying conceptual (as opposed to computational) explanation of this link, which I will place below the fold.