In the field of mathematics known as functional analysis, the invariant subspace problem is a partially unresolved problem asking whether every bounded operator on a complex Banach space sends some non-trivial closed subspace to itself. Many variants of the problem have been solved, by restricting the class of bounded operators considered or by specifying a particular class of Banach spaces. The problem is still open for separable Hilbert spaces (in other words, each example, found so far, of an operator with no non-trivial invariant subspaces is an operator that acts on a Banach space that is not isomorphic to a separable Hilbert space).

The problem seems to have been stated in the mid-20th century after work by Beurling and von Neumann, who found (but never published) a positive solution for the case of compact operators. It was then posed by Paul Halmos for the case of operators ${\displaystyle T}$ such that ${\displaystyle T^{2}}$ is compact. This was resolved affirmatively, for the more general class of polynomially compact operators (operators ${\displaystyle T}$ such that ${\displaystyle p(T)}$ is a compact operator for a suitably chosen nonzero polynomial ${\displaystyle p}$), by Allen R. Bernstein and Abraham Robinson in 1966 (see Non-standard analysis § Invariant subspace problem for a summary of the proof).

For Banach spaces, the first example of an operator without an invariant subspace was constructed by Per Enflo. He proposed a counterexample to the invariant subspace problem in 1975, publishing an outline in 1976. Enflo submitted the full article in 1981 and the article's complexity and length delayed its publication to 1987. Enflo's long "manuscript had a world-wide circulation among mathematicians" and some of its ideas were described in publications besides Enflo (1976). Enflo's works inspired a similar construction of an operator without an invariant subspace for example by Bernard Beauzamy, who acknowledged Enflo's ideas.

In the 1990s, Enflo developed a "constructive" approach to the invariant subspace problem on Hilbert spaces.

In May 2023, a preprint of Enflo appeared on arXiv, which, if correct, solves the problem for Hilbert spaces and completes the picture.

In July 2023, a second and independent preprint of Neville appeared on arXiv, claiming the solution of the problem for separable Hilbert spaces.

In September 2024, a peer-reviewed article published in Axioms by a team of four Jordanian academic researchers announced that they had solved the invariant subspace problem. However, basic mistakes in the proof were pointed out.

Formally, the invariant subspace problem for a complex Banach space ${\displaystyle H}$ of dimension > 1 is the question whether every bounded linear operator ${\displaystyle T:H\to H}$ has a non-trivial closed ${\displaystyle T}$-invariant subspace: a closed linear subspace ${\displaystyle W}$ of ${\displaystyle H}$, which is different from ${\displaystyle \{0\}}$ and from ${\displaystyle H}$, such that ${\displaystyle T(W)\subset W}$.