Hybrid incompatibility is a phenomenon in plants and animals, wherein offspring produced by the mating of two different species or populations have reduced viability and/or are less able to reproduce. Examples of hybrids include mules and ligers from the animal world, and subspecies of the Asian rice crop Oryza sativa from the plant world. Multiple models have been developed to explain this phenomenon. Recent research suggests that the source of this incompatibility is largely genetic, as combinations of genes and alleles prove lethal to the hybrid organism. Incompatibility is not solely influenced by genetics, however, and can be affected by environmental factors such as temperature. The genetic underpinnings of hybrid incompatibility may provide insight into factors responsible for evolutionary divergence between species.

Hybrid incompatibility occurs when the offspring of two closely related species are not viable or suffer from infertility. Charles Darwin posited that hybrid incompatibility is not a product of natural selection, stating that the phenomenon is an outcome of the hybridizing species diverging, rather than something that is directly acted upon by selective pressures. The underlying causes of the incompatibility can be varied: earlier research focused on things like changes in ploidy in plants. More recent research has taken advantage of improved molecular techniques and has focused on the effects of genes and alleles in the hybrid and its parents.

The first major breakthrough in the genetic basis of hybrid incompatibility is the Dobzhansky-Muller model, a combination of findings by Theodosius Dobzhansky and Joseph Muller between 1937 and 1942. The model provides an explanation as to why a negative fitness effect like hybrid incompatibility is not selected against. By hypothesizing that the incompatibility arose from alterations at two or more loci, rather than one, the incompatible alleles are in one hybrid individual for the first time rather than throughout the population - thus, hybrids that are infertile can develop while the parent populations remain viable. The negative fitness effects of infertility are not present in the original population. In this way, hybrid infertility contributes in some part to speciation by ensuring that gene flow between diverging species remains limited. Further analysis of the issue has supported this model, although it does not include conspecific genic interactions, a potential factor that more recent research has begun to look in to.

Decades after the research of Dobzhansky and Muller, the specifics of hybrid incompatibility were explored by Jerry Coyne and H. Allen Orr. Using introgression techniques to analyze the fertility in Drosophila hybrid and non-hybrid offspring, specific genes that contribute to sterility were identified; a study by Chung-I Wu which expanded on Coyne and Orr's work found that the hybrids of two Drosophila species were made sterile by the interaction of around 100 genes. These studies widened the scope of the Dobzhansky-Muller model, who thought it likely that more than two genes would be responsible. The ubiquity of Drosophila as a model organism has allowed many of the sterility genes to be sequenced in the years since Wu's study.^{[citation needed]}

With modern molecular techniques, researchers have been able to more accurately identify the underlying genetic causes of hybrid incompatibility. This has led to both the development of expansions to the Dobzhansky-Muller model. Recent research has also explored the possibility of external influences on sterility as well.

An extension of the Dobzhansky-Muller model is the "snowball effect"; an accumulation of incompatible loci due to increased species divergence. Since the model posits that sterility is due to negative allelic interaction between the hybridizing species, as species become more diverged it follows that more negative interactions should develop. The snowball effect states that the number of these incompatibilities will increase exponentially over the time of divergence, particularly when more than two loci contribute to the incompatibility. This concept has been exhibited in tests with the flowering plant genus Solanum, with the findings supporting the genetic underpinnings of Dobzhansky-Muller:

"Overall, our results indicate that the accumulation of sterility loci follows a different trajectory from the accumulation of loci for other quantitative species differences, consistent with the unique genetic basis expected to underpin species reproductive isolating barriers. ...In doing so, we uncover direct empirical support for the Dobzhansky-Muller model of hybrid incompatibility, and the snowball prediction in particular."

Though the primary causes of hybrid incompatibility appear to be genetic, external factors may play a role as well. Studies focused primarily on model plants have found that the viability of hybrids can be dependent on environmental influence. Several studies on rice and Arabidopsis species identify temperature as an important factor in hybrid viability; generally, low temperatures seem to cause negative hybrid symptoms to be expressed while high temperatures suppress them, although one rice study found the opposite to be true. There has also been evidence in an Arabidopsis species that in poor environmental conditions (in this case, high temperatures), hybrids did not express negative symptoms and are viable with other populations. When environmental conditions return to normal, however, the negative symptoms are expressed and the hybrids are once again incompatible with other populations.