The problem comprises drawing lines from two points, meeting at a third point on the circumference (boundary) of a circle and making equal angles with the normal at that point (specular reflection). It belongs to geometrical optics (in which light is modeled using rays rather than waves or particles), and catoptrics, the use of mirrors to control light: it can be used to find the path of a ray of light that starts at one point of space, is reflected from a spherical mirror, and passes through a second point. Although this is a three-dimensional problem, it can immediately be reduced to the two-dimensional problem of reflection in a circular mirror in the plane, because its solution lies entirely within the plane formed by the two points and the center of the sphere.^[1]

The same problem can be formulated with the two given points inside the circle instead of outside.^[1] For two points near each other within the circle, in general position, there will be two solutions, but points that are farther apart have four solutions.^[2] Any solution describes the path of a billiards ball reflected within a circular billiards table,^[3][4] as Lewis Carroll once suggested for billiards play.^[5] If the two segments of the reflection path are extended to chords of the circle, the two chords make equal angles to the circle and therefore have equal length. Thus, these chords form the two equal sides of an isosceles triangle inscribed within the circle, with the two given points on two sides of this triangle. Another equivalent form of Alhazen's problem asks to construct a triangle with these properties.^[3][6]

Another way of describing the problem, for points inside or outside the circle, is that it seeks an ellipse having the two given points as its foci, tangent to the given circle. The point of tangency is the solution point to Alhazen's problem. A ray from one focus of the ellipse to this point of tangency will be reflected by the ellipse to the other focus (focus-to-focus reflection property). Furthermore, because the given circle has the same angle at the point of tangency, it will reflect the same ray in the same way. In an ellipse, all single-reflection paths from one focus to another have equal lengths, so the smallest ellipse tangent to the circle produces the shortest path from one given point to the circle and then to the other point.^[2][7] The idea that light rays follow shortest paths is Hero's principle, later reformulated in Fermat's principle that light rays follow quickest paths.^[8] More generally, as James Gregory observed, for any analogous three-dimensional reflection problem, the point of reflection is also a point of tangency of an ellipsoid having the source and destination of the reflected ray as its foci.^[9]

Ptolemy included the problem of reflection in a circular mirror in his Optics (written in the second century CE), but was only able to solve certain special cases;^[10][11] al-Haytham formulated and solved the problem more generally.^[10] Al-Haytham was inspired by Ptolemy's work, and modeled his own book on Ptolemy's, but differed from it in important ways; for instance, Ptolemy used a model of visual perception in which visual rays travel outward from the eye to the objects it sees, while al-Haytham reversed this to the still-used model in which light rays travel inward from objects to the eye.^[12][13]

By the time of Pappus of Alexandria, in the 4th century AD, Greek mathematicians had categorized geometric solutions into three types: straightedge and compass constructions, constructions using conic sections, and neusis constructions involving a marked ruler, preferring the earlier categories of solution over the later ones.^[14] Ibn al-Haytham's solution is of the second type, using hyperbola, through which he develops a neusis construction.^[15][16] In his 1881 survey of the problem, Marcus Baker calls al-Haytham's solution "excessively prolix and intricate", and quotes Isaac Barrow as expressing a similar opinion.^[17] Later in the 11th century, Yusuf al-Mu'taman ibn Hud, a king of the Taifa of Zaragoza in Spain, simplified al-Haytham's lemmas somewhat, but did not make a significant advance on the problem.^[18] The work of al-Haytham became known in the rest of Europe through manuscript Latin translations in the 12th or 13th century, and a translation was published in Basel in 1572.^[15] Later geometric solutions by Christiaan Huygens, René-François de Sluse, and Guillaume de l'Hôpital used the same idea of an auxiliary conic section: a hyperbola for Huygens, a parabola for Sluse,^[17][19][20] and both methods for l'Hôpital. Baker cites Huygens's solution as "the most elegant the problem has ever received".^[17]

In al-Haytham's solution, the hyperbola is used within a construction of the angle of reflection, after which the point of reflection is easy to find. Al-Haytham further subdivides the problem into cases, according to the number of reflected images (one for a convex mirror but up to four for a concave mirror), and solves each case separately.^[15][16] Instead, Huygens finds a hyperbola that directly solves the problem in all cases: the reflection points are points of intersection between this hyperbola and the given circle. This hyperbola can be characterized in many ways; one way involves inversive geometry.^[21] The locus of points ${\displaystyle L}$ at which the two lines to the given points cross, at equal angles, a circle concentric to the given one, is a cubic curve containing both given points. The inversion of ${\displaystyle L}$ through the given circle^[a] is a rectangular hyperbola passing through the two points inverse to the given points and centered at the midpoint of the two inverse points. Its asymptotic lines are parallel to and perpendicular to the angle bisector of the angle subtended by the given points (or their inverses) at the center of the circle. The intersections of this hyperbola with the given circle include the desired solution point or points.^[21][22]

The number of solutions, for points inside the circle, can also be determined geometrically. In general terms, pairs of given points that are near each other within the circle, and near to the center of the circle, have two reflection points; pairs of points that are far apart and far from the center have four reflections. If one given point is fixed, the positions of the other point that produce two reflections are separated from the positions that produce four reflections by the caustic generated by the reflections of a light source at the fixed point. On the caustic itself, away from its cusps, there are three reflections. At the cusps there are only two reflections.^[2]

The impossibility of a straightedge and compass solution was finally proven in 1965, using algebraic methods, by Jack M. Elkin (an actuary).^[23][10] A similar impossibility proof was rediscovered in 1997 by Oxford mathematician Peter M. Neumann.^[1][24] The neusis construction can also be carried out using origami folds following the Huzita–Hatori axioms,^[25][26] and Roger C. Alperin has argued through algebraic methods that the problem can be solved by straightedge, compass, and angle trisector, but without providing an explicit construction.^[27]

According to Roberto Marcolongo, a mechanical solution was presented by Leonardo da Vinci, after he failed to find a mathematical solution. The solution, as reconstructed by Marcolongo, takes the form of a mechanical linkage that, when placed with its tip pinned to the circle center and the two given points allowed to slide along its arms, always maintains equal angles to these points at the hinge point of the two arms. Therefore, if the mechanism is moved in order to place this hinge point on the given circle, the solution will be obtained at this point.^{[28][29][30][31]}

Later mathematicians such as James Gregory and many others attempted to find an algebraic solution to the problem, using various methods, including analytic methods of geometry and derivation by complex numbers.^[10][32][17] An algebraic solution to the problem was finally found in 1965 by Elkin, by means of a quartic polynomial.^[23] Other solutions were rediscovered later: in 1989, by Harald Riede;^[33] in 1990 (submitted in 1988), by Miller and Vegh;^[34] and in 1992, by John D. Smith^[10] and also by Jörg Waldvogel.^[22]

Waldvogel simplifies the algebra by formulating the problem for the unit circle and two given points ${\displaystyle a}$ and ${\displaystyle b}$ in the complex plane. With the aid of Huygens's hyperbola Waldvogel derives a quartic equation for the reflection point ${\displaystyle \zeta }$, $${\displaystyle {\overline {ab}}\zeta ^{4}-({\overline {a}}+{\overline {b}})\zeta ^{3}+(a+b)\zeta -ab=0,}$$ where ${\displaystyle {\overline {a}}}$ and ${\displaystyle {\overline {b}}}$ are the complex conjugates of ${\displaystyle a}$ and ${\displaystyle b}$.^[22] Some roots of this equation might not lie on the unit circle, or fail to give a valid reflection path, but the valid solutions can all be found among the roots. The root on the unit circle minimizing the total distance to the given points, ${\displaystyle |a-\zeta |+|\zeta -b|}$, is always a valid solution.^[32] With some further manipulation this can be reformulated as an equation involving real numbers instead of complex numbers.^[22] Elkin instead formulates the problem in terms of the squared Euclidean distances among the given points and the center of the given unit circle, and finds an "inelegant, asymmetrical" quartic equation for the squared distance from one given point to the reflection point, with combinations of the other squared distances as its coefficients.^[23]

The algebraic solution of this problem allows the use of Galois theory to prove that, for certain easily constructed inputs, the solution point has coordinates that are not constructible numbers, and therefore that the problem has no straightedge and compass solution.^[7][23][35] Elkin considers two given points with squared distances ${\displaystyle {\tfrac {1}{2}}}$ and ${\displaystyle {\tfrac {1}{3}}}$ from the center of a unit circle and ${\displaystyle {\tfrac {1}{2}}}$ from each other, and uses the resolvent cubic of his algebraic solution to show that it is not constructible.^[23] Alternatively, Carréga & Haddad (2016) show that the two given points ${\displaystyle (-{\tfrac {1}{10}},-{\tfrac {1}{10}})}$ and ${\displaystyle ({\tfrac {1}{6}},-{\tfrac {1}{6}})}$ have reflection points on the unit circle whose coordinates come from roots of the polynomial ${\displaystyle x^{4}-x-1}$, which has the symmetric group on four elements as its Galois group. It follows that the reflection point cannot be constructed with straightedge and compass. Carréga and Haddad generalize this example to pairs of rational points whose reflection points give the roots of any Stewart polynomial ${\displaystyle x^{4}-rx-1}$ and show that this leads to unconstructible points whenever ${\displaystyle r}$ is a prime number.^[35]

Isaac Barrow, in a set of lectures in 1669, used a trigonometric equation in polar coordinates to describe the cubic curve ${\displaystyle L}$ inverse to Huygens's hyperbola. It has the same property as the hyperbola: it intersects the given circle at the reflection point or points that solve the problem.^[10]

Gander & Gruntz (1992) derive a messy equation involving square roots of polynomials of trigonometric functions for the angle around the circle of the reflection point. They suggest the use of Newton's method to solve this equation numerically, but this involves the derivative of one side of the equation, and they write that finding this derivative explicitly is "certainly not the way to go". Instead, for this part they use automatic differentiation.^[36]

For inputs at which the segment from one given point to the reflection point is perpendicular to the segment between the other given point and the circle center, the Huygens hyperbola degenerates to two lines. This degeneracy can lead certain numerical solutions of Alhazen's problem to become unstable near these inputs.^[37] Additional forms of instability can arise when the circle radius is much smaller than its distance to the two given points. To avoid these issues, Miller, Barnes & MacKenzie (2021) reformulate the problem as a trigonometric equation with, in general, 16 solutions, but for which it is possible to predetermine which of these solutions is the desired one. This leads to an iterative numerical solution that is fast and robust, which they apply in planetary science to modeling light reflection off the lakes of Titan.^[38]

An iterative numerical solution is also possible based on either the law of reflection or Fermat's principle.^[39]

Certain special cases admit simpler solutions. For two given points equidistant from the center of the circle, the reflection point or points occur where the circle is crossed by the perpendicular bisector of the two points. And for two given points that lie on a single diameter of the circle, there are one or two reflection points where this diameter crosses the circle.

As well, when the two points on the diameter are interior to the circle, there may be two more reflection points, where the given circle is crossed by an Apollonian circle through the center of the circle. This circle is the locus of points whose ratio of distances to the two given points is constant. If the Apollonian circle crosses the given circle, reflection points occur at the crossings; however, it may remain entirely within the given circle, in which case the only reflection points are the endpoints of the diameter.^[23]

When the distance to the circle is much greater for one of the given points that it can be assumed practically infinite, a one-finite solution exists whose correct root branch can be determined a priori.^[38]

As well as spherical mirrors, al-Haytham also studied conical and cylindrical mirrors,^[15] which can be reduced in the same way as a spherical mirror to reflection in a circular mirror in the plane.^[37] Researchers have extended Alhazen's problem to general rotationally symmetric quadric mirrors, including hyperbolic, parabolic and elliptical mirrors.^[40] They showed that the mirror reflection point can be computed by solving an eighth-degree equation in the most general case. If the camera (eye) is placed on the axis of the mirror, the degree of the equation reduces to six.^[41]

Alhazen's problem can also be extended to multiple refractions from a spherical ball. Given a light source and a spherical ball of certain refractive index, the closest point on the spherical ball where the light is refracted to the eye of the observer can be obtained by solving a tenth-degree equation.^[41]

Another direction for generalization is to non-Euclidean geometry. In the hyperbolic plane, as in the Euclidean plane, it is not possible to solve the problem using only a straightedge and compass.^[6]