In representation theory, a branch of mathematics, Engel's theorem states that a finite-dimensional Lie algebra ${\displaystyle {\mathfrak {g}}}$ is a nilpotent Lie algebra if and only if for each ${\displaystyle X\in {\mathfrak {g}}}$, the adjoint map

given by ${\displaystyle \operatorname {ad} (X)(Y)=[X,Y]}$, is a nilpotent endomorphism on ${\displaystyle {\mathfrak {g}}}$; i.e., ${\displaystyle \operatorname {ad} (X)^{k}=0}$ for some k. It is a consequence of the theorem, also called Engel's theorem, which says that if a Lie algebra of matrices consists of nilpotent matrices, then the matrices can all be simultaneously brought to a strictly upper triangular form. Note that if we merely have a Lie algebra of matrices which is nilpotent as a Lie algebra, then this conclusion does not follow (i.e. the naïve replacement in Lie's theorem of "solvable" with "nilpotent", and "upper triangular" with "strictly upper triangular", is false; this already fails for the one-dimensional Lie subalgebra of scalar matrices).

The theorem is named after the mathematician Friedrich Engel, who sketched a proof of it in a letter to Wilhelm Killing dated 20 July 1890 (Hawkins 2000, p. 176). Engel's student K.A. Umlauf gave a complete proof in his 1891 dissertation, reprinted as (Umlauf 2010).

Let ${\displaystyle {\mathfrak {gl}}(V)}$ be the Lie algebra of the endomorphisms of a finite-dimensional vector space V and ${\displaystyle {\mathfrak {g}}\subset {\mathfrak {gl}}(V)}$ a subalgebra. Then Engel's theorem states the following are equivalent:

Note that no assumption on the underlying base field is required.

We note that Statement 2. for various ${\displaystyle {\mathfrak {g}}}$ and V is equivalent to the statement

This is the form of the theorem proven in #Proof. (This statement is trivially equivalent to Statement 2 since it allows one to inductively construct a flag with the required property.)

In general, a Lie algebra ${\displaystyle {\mathfrak {g}}}$ is said to be nilpotent if the lower central series of it vanishes in a finite step; i.e., for ${\displaystyle C^{0}{\mathfrak {g}}={\mathfrak {g}},C^{i}{\mathfrak {g}}=[{\mathfrak {g}},C^{i-1}{\mathfrak {g}}]}$ = (i+1)-th power of ${\displaystyle {\mathfrak {g}}}$, there is some k such that ${\displaystyle C^{k}{\mathfrak {g}}=0}$. Then Engel's theorem implies the following theorem (also called Engel's theorem): when ${\displaystyle {\mathfrak {g}}}$ has finite dimension,