The optical transfer function (OTF) of an optical system such as a camera, microscope, human eye, or projector is a scale-dependent description of their imaging contrast. Its magnitude is the image contrast of the harmonic intensity pattern, ${\displaystyle 1+\cos(2\pi \nu \cdot x)}$, as a function of the spatial frequency, ${\displaystyle \nu }$, while its complex argument indicates a phase shift in the periodic pattern. The optical transfer function is used by optical engineers to describe how the optics project light from the object or scene onto a photographic film, detector array, retina, screen, or simply the next item in the optical transmission chain.

Formally, the optical transfer function is defined as the Fourier transform of the point spread function (PSF, that is, the impulse response of the optics, the image of a point source). As a Fourier transform, the OTF is generally complex-valued; however, it is real-valued in the common case of a PSF that is symmetric about its center. In practice, the imaging contrast, as given by the magnitude or modulus of the optical-transfer function, is of primary importance. This derived function is commonly referred to as the modulation transfer function (MTF).

The image on the right shows the optical transfer functions for two different optical systems in panels (a) and (d). The former corresponds to the ideal, diffraction-limited, imaging system with a circular pupil. Its transfer function decreases approximately gradually with spatial frequency until it reaches the diffraction-limit, in this case at 500 cycles per millimeter or a period of 2 μm. Since periodic features as small as this period are captured by this imaging system, it could be said that its resolution is 2 μm. Panel (d) shows an optical system that is out of focus. This leads to a sharp reduction in contrast compared to the diffraction-limited imaging system. It can be seen that the contrast is zero around 250 cycles/mm, or periods of 4 μm. This explains why the images for the out-of-focus system (e,f) are more blurry than those of the diffraction-limited system (b,c). Note that although the out-of-focus system has very low contrast at spatial frequencies around 250 cycles/mm, the contrast at spatial frequencies just below the diffraction limit of 500 cycles/mm is comparable to that of the ideal system. Close observation of the image in panel (f) shows that the image of the large spoke densities near the center of the spoke target is relatively sharp.

Since the optical transfer function (OTF) is defined as the Fourier transform of the point-spread function (PSF), it is generally speaking a complex-valued function of spatial frequency. The projection of a specific periodic pattern is represented by a complex number with absolute value and complex argument proportional to the relative contrast and translation of the projected projection, respectively.

Often the contrast reduction is of most interest and the translation of the pattern can be ignored. The relative contrast is given by the absolute value of the optical transfer function, a function commonly referred to as the modulation transfer function (MTF). Its values indicate how much of the object's contrast is captured in the image as a function of spatial frequency. The MTF tends to decrease with increasing spatial frequency from 1 to 0 (at the diffraction limit); however, the function is often not monotonic. On the other hand, when also the pattern translation is important, the complex argument of the optical transfer function can be depicted as a second real-valued function, commonly referred to as the phase transfer function (PhTF). The complex-valued optical transfer function can be seen as a combination of these two real-valued functions:

where

and ${\displaystyle \mathrm {arg} (\cdot )}$ represents the complex argument function, while ${\displaystyle \nu }$ is the spatial frequency of the periodic pattern. In general ${\displaystyle \nu }$ is a vector with a spatial frequency for each dimension, i.e. it indicates also the direction of the periodic pattern.

The impulse response of a well-focused optical system is a three-dimensional intensity distribution with a maximum at the focal plane, and could thus be measured by recording a stack of images while displacing the detector axially. By consequence, the three-dimensional optical transfer function can be defined as the three-dimensional Fourier transform of the impulse response. Although typically only a one-dimensional, or sometimes a two-dimensional section is used, the three-dimensional optical transfer function can improve the understanding of microscopes such as the structured illumination microscope.