In mathematics, specifically functional analysis, a trace-class operator is a linear operator for which a trace may be defined, such that the trace is a finite number independent of the choice of basis used to compute the trace. This trace of trace-class operators generalizes the trace of matrices studied in linear algebra. All trace-class operators are compact operators.

In quantum mechanics, quantum states are described by density matrices, which are certain trace class operators.

Trace-class operators are essentially the same as nuclear operators, though many authors reserve the term "trace-class operator" for the special case of nuclear operators on Hilbert spaces and use the term "nuclear operator" in more general topological vector spaces (such as Banach spaces).

Let ${\displaystyle H}$ be a separable Hilbert space, ${\displaystyle \left\{e_{k}\right\}_{k=1}^{\infty }}$ an orthonormal basis and ${\displaystyle A:H\to H}$ a positive bounded linear operator on ${\displaystyle H}$. The trace of ${\displaystyle A}$ is denoted by ${\displaystyle \operatorname {Tr} (A)}$ and defined as

independent of the choice of orthonormal basis. A (not necessarily positive) bounded linear operator ${\displaystyle T:H\rightarrow H}$ is called trace class if and only if

where ${\displaystyle |T|:={\sqrt {T^{*}T}}}$ denotes the positive-semidefinite Hermitian square root.

The trace-norm of a trace class operator T is defined as $${\displaystyle \|T\|_{1}:=\operatorname {Tr} (|T|).}$$ One can show that the trace-norm is a norm on the space of all trace class operators ${\displaystyle B_{1}(H)}$ and that ${\displaystyle B_{1}(H)}$, with the trace-norm, becomes a Banach space.

When ${\displaystyle H}$ is finite-dimensional, every (positive) operator is trace class. For ${\displaystyle A}$ this definition coincides with that of the trace of a matrix. If ${\displaystyle H}$ is complex, then ${\displaystyle A}$ is always self-adjoint (i.e. ${\displaystyle A=A^{*}=|A|}$) though the converse is not necessarily true.