In mathematics and theoretical physics, a representation of a Lie group is a linear action of a Lie group on a vector space. Equivalently, a representation is a smooth homomorphism of the group into the group of invertible operators on the vector space. Representations play an important role in the study of continuous symmetry. A great deal is known about such representations, a basic tool in their study being the use of the corresponding 'infinitesimal' representations of Lie algebras.

A complex representation of a group is an action by a group on a finite-dimensional vector space over the field ${\displaystyle \mathbb {C} }$. A representation of the Lie group G, acting on an n-dimensional vector space V over ${\displaystyle \mathbb {C} }$ is then a smooth group homomorphism

where ${\displaystyle \operatorname {GL} (V)}$ is the general linear group of all invertible linear transformations of ${\displaystyle V}$ under their composition. Since all n-dimensional spaces are isomorphic, the group ${\displaystyle \operatorname {GL} (V)}$ can be identified with the group of the invertible, complex ${\displaystyle n\times n}$ matrices, generally called ${\displaystyle \operatorname {GL} (n;\mathbb {C} ).}$ Smoothness of the map ${\displaystyle \Pi }$ can be regarded as a technicality, in that any continuous homomorphism will automatically be smooth.

We can alternatively describe a representation of a Lie group ${\displaystyle G}$ as a linear action of ${\displaystyle G}$ on a vector space ${\displaystyle V}$. Notationally, we would then write ${\displaystyle g\cdot v}$ in place of ${\displaystyle \Pi (g)v}$ for the way a group element ${\displaystyle g\in G}$ acts on the vector ${\displaystyle v\in V}$.

A typical example in which representations arise in physics would be the study of a linear partial differential equation having symmetry group ${\displaystyle G}$. Although the individual solutions of the equation may not be invariant under the action of ${\displaystyle G}$, the space ${\displaystyle V}$ of all solutions is invariant under the action of ${\displaystyle G}$. Thus, ${\displaystyle V}$ constitutes a representation of ${\displaystyle G}$. See the example of SO(3), discussed below.

If the homomorphism ${\displaystyle \Pi }$ is injective (i.e., a monomorphism), the representation is said to be faithful.

If a basis for the complex vector space V is chosen, the representation can be expressed as a homomorphism into general linear group ${\displaystyle \operatorname {GL} (n;\mathbb {C} )}$. This is known as a matrix representation. Two representations of G on vector spaces V, W are equivalent if they have the same matrix representations with respect to some choices of bases for V and W.

Given a representation ${\displaystyle \Pi :G\rightarrow \operatorname {GL} (V)}$, we say that a subspace W of V is an invariant subspace if ${\displaystyle \Pi (g)w\in W}$ for all ${\displaystyle g\in G}$ and ${\displaystyle w\in W}$. The representation is said to be irreducible if the only invariant subspaces of V are the zero space and V itself. For certain types of Lie groups, namely compact and semisimple groups, every finite-dimensional representation decomposes as a direct sum of irreducible representations, a property known as complete reducibility. For such groups, a typical goal of representation theory is to classify all finite-dimensional irreducible representations of the given group, up to isomorphism. (See the Classification section below.)