The last two lectures of this course will be on Ratner’s theorems on equidistribution of orbits on homogeneous spaces. Due to lack of time, I will not be able to cover all the material here that I had originally planned; in particular, for an introduction to this family of results, and its connections with number theory, I will have to refer readers to my previous blog post on these theorems. In this course, I will discuss two special cases of Ratner-type theorems. In this lecture, I will talk about Ratner-type theorems for discrete actions (of the integers ${\Bbb Z})$ on nilmanifolds; this case is much simpler than the general case, because there is a simple criterion in the nilmanifold case to test whether any given orbit is equidistributed or not. Ben Green and I had need recently to develop quantitative versions of such theorems for a number-theoretic application. In the next and final lecture of this course, I will discuss Ratner-type theorems for actions of $SL_2({\Bbb R})$, which is simpler in a different way (due to the semisimplicity of $SL_2({\Bbb R})$, and lack of compact factors).

— Nilpotent groups —

Before we can get to Ratner-type theorems for nilmanifolds, we will need to set up some basic theory for these nilmanifolds. We begin with a quick review of the concept of a nilpotent group – a generalisation of that of an abelian group. Our discussion here will be purely algebraic (no manifolds, topology, or dynamics will appear at this stage).

Definition 1. (Commutators) Let G be a (multiplicative) group. For any two elements g,h in G, we define the commutator [g,h] to be ${}[g,h] := g^{-1}h^{-1}gh$ (thus g and h commute if and only if the commutator is trivial). If H and K are subgroups of G, we define the commutator ${}[H,K]$ to be the group generated by all the commutators $\{ [h,k]: h \in H, k \in K \}$.

For future reference we record some trivial identities regarding commutators:

$gh = hg[g,h] = [g^{-1},h^{-1}]hg$ (1)

$h^{-1} g h = g [g,h] = [h,g^{-1}] g$ (2)

${}[g,h]^{-1} = [h,g]$. (3)

Exercise 1. Let H, K be subgroups of a group G.

1. Show that [H,K] = [K,H].
2. Show that H is abelian if and only if [H,H] is trivial.
3. Show that H is central if and only if [H,G] is trivial.
4. Show that H is normal if and only if ${}[H,G] \subset H$.
5. Show that [H,G] is always normal.
6. If $L \lhd H, K$ is a normal subgroup of both H and K, show that ${}[H,K] / ([H,K] \cap L) \equiv [H/L, K/L]$.
7. Let HK be the group generated by $H \cup K$. Show that ${}[H,K]$ is a normal subgroup of HK, and when one quotients by this subgroup, the images of H and K are groups that commute with each other. $\diamond$

Exercise 2. Let G be a group. Show that the group G/[G,G] is abelian, and is the universal abelianisation of G in the sense that every homomorphism $\phi: G \to H$ from G to an abelian group H can be uniquely factored as $\phi = \tilde \phi \circ \pi$, where $\pi: G \to G/[G,G]$ is the quotient map and $\tilde \phi: G/[G,G] \to H$ is a homomorphism. $\diamond$

Definition 2. (Nilpotency) Given any group G, define the lower central series

$G = G_0 = G_1 \rhd G_2 \rhd G_3 \rhd \ldots$ (4)

by setting $G_0, G_1 := G$ and $G_{i+1} := [G_i,G]$ for $i \geq 1$. We say that G is nilpotent of step s if $G_{s+1}$ is trivial (and $G_s$ is non-trivial).

Examples 1. A group is nilpotent of step 0 if and only if it is trivial. It is nilpotent of step 1 if and only if it is non-trivial and abelian. Any subgroup or homomorphic image of a nilpotent group of step s is nilpotent of step at most s. The direct product of two nilpotent groups is again nilpotent, but the semi-direct product of nilpotent groups is merely solvable in general. If G is any group, then $G/G_{s+1}$ is nilpotent of step at most s. $\diamond$

Example 2. Let $n \geq 1$ be an integer, and let

$U_n({\Bbb R)} = \begin{pmatrix} 1 & {\Bbb R} & \ldots & {\Bbb R} \\ 0 & 1 & \ldots & {\Bbb R} \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & 1 \end{pmatrix}$ (5)

be the group of all upper-triangular $n \times n$ real matrices with $1$s on the diagonal (i.e. the group of unipotent upper-triangular matrices). Then $U_n({\Bbb R})$ is nilpotent of step n. Similarly if ${\Bbb R}$ is replaced by other fields. $\diamond$

Exercise 3. Let G be an arbitrary group.

1. Show that each element $G_i$ of the lower central series is a characteristic subgroup of G, i.e. $\phi(G_i)=G_i$ for all automorphisms $\phi: G \to G$. (Specialising to inner automorphisms, this shows that the $G_i$ are all normal subgroups of G.)
2. Show the filtration property ${}[G_i,G_j] \subset G_{i+j}$ for all $i,j \geq 0$. (Hint: induct on i+j; then, holding i+j fixed, quotient by $G_{i+j}$, and induct on (say) i. Note that once one quotients by $G_{i+j}$, all elements of ${}[G_{i-1},G_j]$ are central (by the first induction hypothesis), while $G_{i-1}$ commutes with ${}[G,G_j]$ (by the second induction hypothesis). Use these facts to show that all the generators of ${}[G,G_{i-1}]$ commute with $G_j$.) $\diamond$

Exercise 4. Let G be a nilpotent group of step 2. Establish the identity

$g^n h^n = (gh)^n [g,h]^{\binom{n}{2}}$ (6)

for any integer n and any $g,h \in G$, where $\binom{n}{2} := \frac{n(n-1)}{2}$. (This can be viewed as a discrete version of the first two terms of the Baker-Campbell-Hausdorff formula.) Conclude in particular that the space of Hall-Petresco sequences $n \mapsto g_0 g_1^n g_2^{\binom{n}{2}}$, where $g_i \in G_i$ for $i=0,1,2$, is a group under pointwise multiplication (this group is known as the Hall-Petresco group of G). There is an analogous identity (and an analogous group) for nilpotent groups of higher step; see for instance this paper of Leibman for details. The Hall-Petresco group is rather useful for understanding multiple recurrence and polynomial behaviour in nilmanifolds; we will not discuss this in detail, but see Exercise 5 below for a hint as to the connection. $\diamond$

Exercise 5. (Arithmetic progressions in nilspaces are constrained) Let X be a nilspace of step $s \leq 2$, and consider two arithmetic progressions $x, gx, \ldots g^{s+1}x$ and $y, hy,\ldots,h^{s+1} y$ of length s+2 in X, where $x,y \in X$ and $g, h \in G$. Show that if these progressions agree in the first s+1 places (thus $g^i x = h^i y$ for all $i=0,\dots,s$ then they also agree in the last place. (Hint: the only tricky case is s=2. For this, either use direct algebraic computation, or experiment with the group of Hall-Petresco sequences from the previous exercise. The claim is in fact true for general s, because the Hall-Petresco group exists in every step.) $\diamond$

Remark 1. By Exercise 3.2, the lower central series is a filtration with respect to the commutator operation $g, h \mapsto [g,h]$. Conversely, if G admits a filtration $G = G_{(0)} = G_{(1)} \geq \ldots$ with ${}[G_{(i)},G_{(j)}] \subset G_{(i+j)}$ and $G_{(j)}$ trivial for $j > s$, then it is nilpotent of step at most s. It is sometimes convenient for inductive purposes to work with filtrations rather than the lower central series (which is the “minimal” filtration available to a group G); see for instance my paper with Ben Green on this topic. $\diamond$

Remark 2. Let G be a nilpotent group of step s. Then ${}[G,G_s] = G_{s+1}$ is trivial and so $G_s$ is central (by Exercise 1), thus abelian and normal. By another application of Exercise 1, we see that $G/G_s$ is nilpotent of step s-1. Thus we see that any nilpotent group G of step s is an abelian extension of a nilpotent group $G/G_s$ of step s-1, in the sense that we have a short exact sequence

$0 \to G_s \to G \to G/G_s \to 0$ (6)

where the kernel $G_s$ is abelian. Conversely, every abelian extension of an s-1-step nilpotent group is nilpotent of step at most s. In principle, this gives a recursive description of s-step nilpotent groups as an s-fold iterated tower of abelian extensions of the trivial group. Unfortunately, while abelian groups are of course very well understood, abelian extensions are a little inconvenient to work with algebraically; the sequence (6) is not quite enough, for instance, to assert that G is a semi-direct product of $G_s$ and $G/G_s$ (this would require some means of embedding $G/G_s$ back into G, which is not available in general). One can identify G (using the axiom of choice) with a product set $G/G_s \times G_s$ with a group law $(g,n) \cdot (h,m) = (gh, nm \phi(g,h))$, where $\phi: G/G_s \times G/G_s \to G_s$ is a map obeying various cocycle-type identities, but the algebraic structure of $\phi$ is not particularly easy to exploit. Nevertheless, this recursive tower of extensions seems to be well suited for understanding the dynamical structure of nilpotent groups and their quotients, as opposed to their algebraic structure (cf. our use of recursive towers of extensions in our previous lectures in dynamical systems and ergodic theory). $\diamond$

In our applications we will not be working with nilpotent groups G directly, but rather with their homogeneous spaces X, i.e. spaces with a transitive left-action of G. (Later we will also add some topological structure to these objects, but let us work in a purely algebraic setting for now.) Such spaces can be identified with group quotients $X \equiv G/\Gamma$ where $\Gamma \leq G$ is the stabiliser $\Gamma = \{ g \in G: gx = x \}$ of some point x in X. (By the transitivity of the action, all stabilisers are conjugate to each other.) It is important to note that in general, $\Gamma$ is not normal, and so X is not a group; it has a left-action of G but not right-action of G. Note though that any central subgroup of G acts on either the left or the right.

Now let G be s-step nilpotent, and let us temporarily refer to $X=G/\Gamma$ as an s-step nilspace. Then G_s acts on the right in a manner that commutes with the left-action of G. If we set $\Gamma_s := G_s \cap \Gamma \lhd G_s$, we see that the right-action of $\Gamma_s$ on $G/\Gamma$ is trivial; thus we in fact have a right-action of the abelian group $T_s := G_s/\Gamma_s$. (In our applications, $T_s$ will be a torus.) This action can be easily verified to be free. If we let $\overline{X} := X/T_s$ be the quotient space, then we can view X as a principal $T_s$-bundle over $\overline{X}$. It is not hard to see (cf. the isomorphism theorems) that $\overline{X} \equiv \pi(G)/\pi(\Gamma)$, where $\pi: G \to G/G_s$ is the quotient map. Observe that $\pi(G)$ is nilpotent of step s-1, and $\pi(\Gamma)$ is a subgroup. Thus we have expressed an arbitrary s-step nilspace as a principal bundle (by some abelian group) over an s-1-step nilspace, and so s-step nilspaces can be viewed as towers of abelian principal bundles, just as s-step nilpotent groups can be viewed as towers of abelian extensions.

— Nilmanifolds —

It is now time to put some topological structure (and in particular, Lie structure) on our nilpotent groups and nilspaces.

Definition 3 (Nilmanifolds). An s-step nilmanifold is a nilspace $G/\Gamma$, where G is a finite-dimensional Lie group which is nilpotent of step s, and $\Gamma$ is a discrete subgroup which is cocompact or uniform in the sense that the quotient $G/\Gamma$ is compact.

Remark 3. In the literature, it is sometimes assumed that the nilmanifold $G/\Gamma$ is connected, and that the group G is connected, or at least that its group $\pi_0(G) := G/G^\circ$ of connected components ($G^\circ \lhd G$ being the identity component of $G$) is finitely generated (one can often easily reduce to this case in applications). It is also convenient to assume that $G^\circ$ is simply connected (again, one can usually reduce to this case in applications, by passing to the universal cover of $G^\circ$ if necessary), as this implies (by the Baker-Campbell-Hausdorff formula) that the nilpotent Lie group $G^\circ$ is exponential, i.e. the exponential map $\exp: {\mathfrak g} \to G^\circ$ is a homeomorphism. $\diamond$

Example 3. (Skew torus) If we define

$G := \begin{pmatrix} 1 & {\Bbb R} & {\Bbb R} \\ 0 & 1 & {\Bbb Z} \\ 0 & 0 & 1 \end{pmatrix}; \quad \Gamma := \begin{pmatrix} 1 & {\Bbb Z} & {\Bbb Z} \\ 0 & 1 & {\Bbb Z} \\ 0 & 0 & 1 \end{pmatrix}$ (7)

(thus G consists of the upper-triangular unipotent matrices whose middle right entry is an integer, and $\Gamma$ is the subgroup in which all entries are integers) then $G/\Gamma$ is a 2-step nilmanifold. If we write

${}[x,y] := \begin{pmatrix} 1 & x & y \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} \Gamma$ (8)

then we see that $G/\Gamma$ is isomorphic to the square $\{ [x,y]: 0 \leq x,y \leq 1 \}$ with the identifications ${}[x,1] \equiv [x,0]$ and ${}[0,y] := [1,y]$. (Topologically, this is homeomorphic to the ordinary 2-torus $({\Bbb R}/{\Bbb Z})^2$, but the skewness will manifest itself when we do dynamics.) $\diamond$

Example 4. (Heisenberg nilmanifold) If we set

$G := \begin{pmatrix} 1 & {\Bbb R} & {\Bbb R} \\ 0 & 1 & {\Bbb R} \\ 0 & 0 & 1 \end{pmatrix}; \quad \Gamma := \begin{pmatrix} 1 & {\Bbb Z} & {\Bbb Z} \\ 0 & 1 & {\Bbb Z} \\ 0 & 0 & 1 \end{pmatrix}$ (9)

then $G/\Gamma$ is a 2-step nilmanifold. It can be viewed as a three-dimensional cube with the faces identified in a somewhat skew fashion, similarly to the skew torus in Example 3. $\diamond$

Let ${\mathfrak g}$ be the Lie algebra of G. Every element g of G acts linearly on ${\mathfrak g}$ by conjugation. Since G is nilpotent, it is not hard to see (by considering the iterated commutators of g with an infinitesimal perturbation of the identity) that this linear action is unipotent, and in particular has determinant 1. Thus, any constant volume form on this Lie algebra will be preserved by conjugation, which by basic differential geometry allows us to create a volume form (and hence a measure) on G which is invariant under both left and right translation; this Haar measure is clearly unique up to scalar multiplication. (In other words, nilpotent Lie groups are unimodular.) Restricting this measure to a fundamental domain of $G/\Gamma$ and then descending to the nilmanifold we obtain a left-invariant Haar measure, which (by compactness) we can normalise to be a Borel probability measure. (Because of the existence of a left-invariant probability measure $\mu$ on $G/\Gamma$, we refer to the discrete subgroup $\Gamma$ of G as a lattice.) One can show that this left-invariant Borel probability measure is unique.

Definition 4. (Nilsystem) An s-step nilsystem (or nilflow) is a topological measure-preserving system (i.e. both a topological dynamical system and a measure-preserving system) with underlying space $G/\Gamma$ a s-step nilmanifold (with the Borel $\sigma$-algebra and left-invariant probability measure), with a shift T of the form $T: x \mapsto g x$ for some $g \in G$.

Example 5. The Kronecker systems $x \mapsto x+\alpha$ on compact abelian Lie groups are 1-step nilsystems. $\diamond$

Example 6. The skew shift system $(x,y) \mapsto (x+\alpha,y+x)$ on the torus $({\Bbb R}/{\Bbb Z})^2$ can be identified with a nilflow on the skew torus (Example 3), after identifying (x,y) with [x,y] and using the group element

$g := \begin{pmatrix} 1 & \alpha & 0 \\ 0 & 1 & 1 \\ 0 & 0 & -1 \end{pmatrix}$ (10)

to create the flow. $\diamond$

Example 7. Consider the Heisenberg nilmanifold (Example 4) with a flow generated by a group element

$g := \begin{pmatrix} 1 & \gamma & \beta \\ 0 & 1 & \alpha \\ 0 & 0 & 1 \end{pmatrix}$ (11)

for some real numbers $\alpha,\beta,\gamma$. If we identify

${}[x,y,z] := \begin{pmatrix} 1 & z & y \\ 0 & 1 & x \\ 0 & 0 & 1 \end{pmatrix} \Gamma$ (12)

then one can verify that

$T^n: [x,y,z] \mapsto [ \{ x+n\alpha\}, y+n\beta + \frac{n(n-1)}{2} \alpha \gamma - \lfloor x$

$+ n\alpha \rfloor(z+n\gamma) \hbox{ mod } 1, z+n\gamma \hbox{ mod } 1]$ (13)

where $\lfloor \rfloor$ and $\{ \}$ are the integer part and fractional part functions respectively. Thus we see that orbits in this nilsystem are vaguely quadratic in n, but for the presence of the not-quite-linear operators $\lfloor \rfloor$ and $\{ \}$. (These expressions are known as bracket polynomials, and are intimately related to the theory of nilsystems.) $\diamond$

Given that we have already seen that nilspaces of step s are principal abelian bundles of nilspaces of step s-1, it should be unsurprising that nilsystems of step s are abelian extensions of nilsystems of step s-1. But in order to ensure that topological structure is preserved correctly, we do need to verify one point:

Lemma 1. Let $G/\Gamma$ be an s-step nilmanifold, with G connected and simply connected. Then $\Gamma_s := G_s \cap \Gamma$ is a discrete cocompact subgroup of $G_s$. In particular, $T_s := G_s/\Gamma_s$ is a compact connected abelian Lie group (in other words, it is a torus).

Proof. Recall that G is exponential and thus identifiable with its Lie algebra ${\mathcal g}$. The commutators $G_i$ can be similarly identified with the Lie algebra commutators ${\mathcal g}_i$; in particular, the $G_i$ are all connected, simply connected Lie groups.

The key point to verify is the cocompact nature of $\Gamma_s$ in $G_s$; all other claims are straightforward. We first work in the abelianisation $G/G_2$, which is identifiable with its Lie algebra and thus isomorphic to a vector space. The image of $\Gamma$ under the quotient map $G \to G/G_2$ is a cocompact subgroup of this vector space; in particular, it contains a basis of this space. This implies that $\Gamma$ contains an “abelianised” basis $e_1,\ldots,e_d$ of G in the sense that every element of G can be expressed in the form $e_1^{t_1} \ldots e_d^{t_d}$ modulo an element of the normal subgroup $G_2$ for some real numbers $t_1,\ldots,t_d$, where we take advantage of the exponential nature of G to define real exponentiation $g^t := \exp( t \log(g) )$. Taking commutators s times (which eliminates all the “modulo $G_2$” errors), we then see that $G_s$ is generated by expressions of the form ${} [e_{i_1},[e_{i_2},[\ldots,e_{i_s}]\ldots]^t$ for $i_1,\ldots,i_s \in \{1,\ldots,d\}$ and real t. Observe that these expressions lie in $\Gamma_s$ if t is integer. As $G_s$ is abelian, we conclude that each element in $G_s$ can be expressed as an element of $\Gamma_s$, times a bounded number of elements of the form ${} [e_{i_1},[e_{i_2},[\ldots,e_{i_s}]\ldots]^t$ with $0 \leq t < 1$. From this we conclude that the quotient map $G_s \mapsto G_s/\Gamma_s$ is already surjective on some bounded set, which we can take to be compact, and so $G_s/\Gamma_s$ is compact as required. $\Box$

As a consequence of this lemma, we see that if $X = G/\Gamma$ is an s-step nilmanifold with G connected and simply connected, then $X/T_s$ is an s-1-step nilmanifold (with G still connected and simply connected), and that X is a principal $T_s$-bundle over $X/T_s$ in the topological sense as well as in the purely algebraic sense. One consequence of this is that every s-step nilsystem (with G connected and simply connected) can be viewed as a toral extension (i.e. a group extension by a torus) of an s-1-step nilsystem (again with G connected and simply connected). Thus for instance the skew shift system (Example 6) is a circle extension of a circle shift, while the Heisenberg nilsystem (Example 7) is a circle extension of an abelian 2-torus shift.

Remark 4. One should caution though that the converse of the above statement is not necessarily true; an extension $X \times_\phi T$ of an s-1-step nilsystem X by a torus T using a cocycle $\phi: X \to T$ need not be isomorphic to an s-step nilsystem (the cocycle $\phi$ has to obey an an additional equation (or more precisely, a system of equations when $s > 2$), known as the Conze-Lesigne equation, before this is the case. See for instance this paper of Ziegler for further discussion. $\diamond$

Exercise 6. Show that Lemma 1 continues to hold if we relax the condition that G is connected and simply connected, to instead require that $G/\Gamma$ is connected, that $G/G^\circ$ is finitely generated, and that $G^\circ$ is simply connected. (I believe that all three of these hypotheses are necessary, but haven’t checked this carefully.) $\diamond$

Exercise 7. Show that Lemma 1 continues to hold if $G_s$ and $\Gamma_s$ are replaced by $G_i$ and $\Gamma_i = G_i \cap \Gamma$ for any $0 \leq i \leq s$. In particular, setting i=2, we obtain a projection map $\pi: X \to X_2$ from X to the Kronecker nilmanifold $X_2 = (G/G_2)/(\Gamma G_2/G_2)$. $\diamond$

Remark 5. One can take the structural theory of nilmanifolds much further, in particular developing the theory of Mal’cev bases (of which the elements $e_1,\ldots,e_d$ used to prove Lemma 1 were a very crude prototype). See the foundational paper of Mal’cev (or its English translation) for details, as well as the later paper of Leibman which addresses the case in which G is not necessarily connected. $\diamond$

— A criterion for ergodicity —

We now give a useful criterion to determine when a given nilsystem is ergodic.

Theorem 1. Let $(X,T) = (G/\Gamma, x \mapsto gx)$ be an s-step nilsystem with G connected and simply connected, and let $(X_2, T_2)$ be the underlying Kronecker factor, as defined in Exercise 7. Then X is ergodic if and only if $X_2$ is ergodic.

This result is originally due to Leon Green, using spectral theory methods. We will use an argument of Parry (and adapted by Leibman), relying on “vertical” Fourier analysis and topological arguments, which we have already used for the skew shift in Proposition 1 of Lecture 9.

Proof. If X is ergodic, then the factor $X_2$ is certainly ergodic. To prove the converse implication, we induct on s. The case $s \leq 1$ is trivial, so suppose $s>1$ and the claim has already been proven for s-1. Then if $X_2$ is ergodic, we already know from induction hypothesis that $X/T_s$ is ergodic. Suppose for contradiction that X is not ergodic, then we can find a non-constant shift-invariant function on X. Using Fourier analysis (or representation theory) of the vertical torus $T_s$ as in Proposition 1 of Lecture 9, we may thus find a non-constant shift-invariant function f which has a single vertical frequency $\chi$ in the sense that one has $f( g_s x ) = \chi(g_s) f(x)$ for all $x \in X$, $g_s \in G_s$, and some character $\chi: G_s \to S^1$. If the character $\chi$ is trivial, then f descends to a non-constant shift-invariant function on $X/T_s$, contradicting the ergodicity there, so we may assume that $\chi$ is non-trivial. Also, |f| descends to a shift-invariant function on $X/T_s$ and is thus constant by ergodicity; by normalising we may assume $|f|=1$.

Now let $g_{s-1} \in G_{s-1}$, and consider the function $F_{g_{s-1}}(x) := f( g_{s-1} x) \overline{f(x)}$. As $G_s$ is central, we see that $F_{g_{s-1}}$ is $G_s$-invariant and thus descends to $X/T_s$. Furthermore, as f is shift-invariant (so $f(g x) = f(x)$), and ${}[g_{s-1},g] \in G_s$, some computation reveals that $F_{g_{s-1}}$ is an eigenfunction:

$F_{g_{s-1}}( g x ) = \chi([g_{s-1},g]) F_{g_{s-1}}(x)$. (14)

In particular, if $\chi([g_{s-1},g]) \neq 1$, then $F_{g_{s-1}}$ must have mean zero. On the other hand, by continuity (and the fact that |f|=1) we know that $F_{g_{s-1}}$ has non-zero mean for $g_{s-1}$ close enough to the identity. We conclude that $\chi([g_{s-1},g]) = 1$ for all $g_{s-1}$ close to the identity; as the map $g_{s-1} \mapsto \chi([g_{s-1},g])$ is a homomorphism, we conclude in fact that $\chi([g_{s-1},g]) = 1$ for all $g_{s-1}$. In particular, from (14) and ergodicity we see that $F_{g_{s-1}}$ is constant, and so $f(g_{s-1} x) = c(g_{s-1}) f(x)$ for some $c(g_{s-1}) \in S^1$.

Now let $h \in G$ be arbitrary. Observe that

$\int_G f(h g_{s-1} x) \overline{f(x)}\ d\mu = \int_G f(h y) \overline{f(g_{s-1}^{-1} y)}\ d\mu$

$= c(g_{s-1}) \int_G f(hy) \overline{f(y)}\ d\mu$

$= \int_G f(g_{s-1} hy) \overline{f(y)}\ d\mu$

$= \chi([g_{s-1},h]) \int_G f(h g_{s-1} y) \overline{f(y)}\ d\mu$. (15)

For h and $g_{s-1}$ close enough to the identity, the integral is non-zero, and we conclude that $\chi([g_{s-1},h])=1$ in this case. The map $(g_{s-1},h) \mapsto \chi([g_{s-1},h])$ is a homomorphism in each variable and so is constant. Since $G_s = [G_{s-1},G]$, we conclude that $\chi$ is trivial, a contradiction. $\Box$

Remark 6. The hypothesis that G is connected and simply connected can be dropped; see the paper of Leibman for details. $\diamond$

One pleasant fact about nilsystems, as compared with arbitrary dynamical systems, is that ergodicity can automatically be upgraded to unique ergodicity:

Theorem 2. Let (X,T) be an ergodic nilsystem. Then (X,T) is also uniquely ergodic. Equivalently, for every $x \in X$, the orbit $(T^n x)_{n \in {\Bbb Z}}$ is equidistributed.

Exercise 8. By inducting on step and adapting the proof of Proposition 3 from Lecture 9, prove Theorem 2. $\diamond$

— a Ratner-type theorem —

A subnilsystem of a nilsystem $(X,T) = (G/\Gamma,T)$ is a compact subsystem $(Y,S)$ which is of the form $Y = Hx$ for some $x \in X$ and some closed subgroup $H \leq G$. One easily verifies that a subnilsystem is indeed a nilsystem.

From the above theorems we quickly obtain

Corollary 1 (Dichotomy between structure and randomness) Let (X,T) be a nilsystem with group G connected and simply connected, and let $x \in X$. Then exactly one of the following statements is true:

1. The orbit $(T^n x)_{n \in {\Bbb Z}}$ is equidistributed.
2. The orbit $(T^n x)_{n \in {\Bbb Z}}$ is contained in a proper subnilsystem (Y,S) with group H connected and simply connected, and with dimension strictly smaller than that of G.

Proof. It is clear that 1. and 2. cannot both be true. Now suppose that 1. is false. By Theorem 2, this means that (X,T) is not ergodic; by Theorem 1, this implies that the Kronecker system $(X_2,T_2)$ is not ergodic. Expanding functions on $X_2 \equiv G/G_2$ into characters and using Fourier analysis, we conclude that there is a non-trivial character $\chi: G/G_2 \to S^1$ which is $T_2$-invariant. If we let $\pi: G \to G/G_2$ be the canonical projection, then $\chi: G \to S^1$ is a continuous homomorphism, and the kernel H is a closed connected subgroup of G of strictly lower dimension. Furthermore, Hx is equal to a level set of $\chi$ and is thus compact. Since $\chi$ is $T_2$ invariant, we see that $T^n x \in Hx$ for all n, and the claim follows. $\Box$

Iterating this, we obtain

Corollary 2 (Ratner-type theorem for nilmanifolds) Let (X,T) be a nilsystem with group G connected and simply connected, and let $x \in X$. Then the orbit $(T^n x)_{n \in {\Bbb Z}}$ is equidistributed in some subnilmanifold (Y,S) of (X,T). (In particular, this orbit is dense in Y.) Furthermore, Y= Hx for some closed connected subgroup H of G.

Remark 7. Analogous claims also hold when G is not assumed to be connected or simply connected, and if the orbit $(T^n x)_{n \in {\Bbb Z}}$ is replaced with a polynomial orbit $(T^{p(n)} x)_{n \in {\Bbb Z}}$; see this paper of Leibman for details (and this followup paper for the case of ${\Bbb Z}^d$-actions. In a different direction, such discrete Ratner-type theorems have been extended to other unipotent actions on finite volume homogeneous spaces by Shah. Quantitative versions of this theorem have also been obtained by Ben Green and myself. $\diamond$

[Update, June 29: minor corrections.]