In mathematics, a Kleinian group is a discrete subgroup of the group of orientation-preserving isometries of hyperbolic 3-space H³. The latter, identifiable with PSL(2, C), is the quotient group of the 2 by 2 complex matrices of determinant 1 by their center, which consists of the identity matrix and its product by −1. PSL(2, C) has a natural representation as orientation-preserving conformal transformations of the Riemann sphere, and as orientation-preserving conformal transformations of the open unit ball B³ in R³. The group of Möbius transformations is also related as the non-orientation-preserving isometry group of H³, PGL(2, C). So, a Kleinian group can be regarded as a discrete subgroup acting on one of these spaces.

The theory of general Kleinian groups was founded by Felix Klein (1883) and Henri Poincaré (1883), who named them after Felix Klein. The special case of Schottky groups had been studied a few years earlier, in 1877, by Friedrich Schottky.

One modern definition of Kleinian group is as a group which acts on the 3-ball ${\displaystyle B^{3}}$ as a discrete group of hyperbolic isometries. Hyperbolic 3-space has a natural boundary; in the ball model, this can be identified with the 2-sphere. We call it the sphere at infinity, and denote it by ${\displaystyle S_{\infty }^{2}}$. A hyperbolic isometry extends to a conformal homeomorphism of the sphere at infinity (and conversely, every conformal homeomorphism on the sphere at infinity extends uniquely to a hyperbolic isometry on the ball by Poincaré extension). It is a standard result from complex analysis that conformal homeomorphisms on the Riemann sphere are exactly the Möbius transformations, which can further be identified as elements of the projective linear group PGL(2,C). Thus, a Kleinian group can also be defined as a subgroup Γ of PGL(2,C). Classically, a Kleinian group was required to act properly discontinuously on a non-empty open subset of the Riemann sphere, but modern usage allows any discrete subgroup.

When Γ is isomorphic to the fundamental group ${\displaystyle \pi _{1}}$ of a hyperbolic 3-manifold, then the quotient space H³/Γ becomes a Kleinian model of the manifold. Many authors^[who?] use the terms Kleinian model and Kleinian group interchangeably, letting the one stand for the other.

Discreteness implies points in the interior of hyperbolic 3-space have finite stabilizers, and discrete orbits under the group Γ. On the other hand, the orbit Γp of a point p will typically accumulate on the boundary of the closed ball ${\displaystyle {\bar {B}}^{3}}$.

The set of accumulation points of Γp in ${\displaystyle S_{\infty }^{2}}$ is called the limit set of Γ, and usually denoted ${\displaystyle \Lambda (\Gamma )}$. The complement ${\displaystyle \Omega (\Gamma )=S_{\infty }^{2}-\Lambda (\Gamma )}$ is called the domain of discontinuity or the ordinary set or the regular set. Ahlfors' finiteness theorem implies that if the group is finitely generated then ${\displaystyle \Omega (\Gamma )/\Gamma }$ is a Riemann surface orbifold of finite type.

The unit ball B³ with its conformal structure is the Poincaré model of hyperbolic 3-space. When we think of it metrically, with metric

it is a model of 3-dimensional hyperbolic space H³. The set of conformal self-maps of B³ becomes the set of isometries (i.e. distance-preserving maps) of H³ under this identification. Such maps restrict to conformal self-maps of ${\displaystyle S_{\infty }^{2}}$, which are Möbius transformations. There are isomorphisms