Inbreeding (i.e., breeding between closely related individuals) results in more recessive traits manifesting themselves, as the genomes of pair-mates are more similar. Recessive traits can only occur in an offspring if present in both parents' genomes. The more genetically similar the parents are, the more often recessive traits appear in their offspring. This normally has a positive effect, as most genes are undergoing purifying selection (the homozygous state is favored). However, for very closely related individuals, there is an increased likelihood of homozygous deleterious genes in the offspring which can result in less-fit individuals.^[4] For the alleles that confer an advantage in the heterozygous and/or homozygous-dominant state, the fitness of the homozygous-recessive state may even be zero (meaning sterile or unviable offspring).

In inbred populations, especially small ones, genetic drift causes alleles to become fixed in a population.^[5] Drift load is the decline in fitness in a population due to the fixation of deleterious alleles.^[6][7]

An example of inbreeding depression is shown in the image. In this case, a is the recessive allele which has negative effects. In order for the a phenotype to become active, the gene must end up as homozygous aa because in the geneotype Aa, the A takes dominance over the a and the a does not have any effect. Some recessive genes result in detrimental phenotypes by causing the organism to be less fit to its natural environment.

Another mechanism responsible for inbreeding depression is the fitness advantage of heterozygosity, which is known as overdominance. This can lead to reduced fitness of a population with many homozygous genotypes, even if they are not deleterious or recessive. Here, even the dominant alleles result in reduced fitness if present homozygously (see also hybrid vigour).

Overdominance is rare in nature.^[4] For practical applications, e.g. in livestock breeding, the former is thought to be more significant – it may yield completely unviable offspring (meaning outright failure of a pedigree), while the latter can only result in relatively reduced fitness.

Natural selection cannot effectively remove all deleterious recessive genes from a population for several reasons. First, deleterious genes arise constantly through de novo mutation within a population. Second, most offspring will have some deleterious traits, so few will be more fit for survival than the others. Different deleterious traits are extremely unlikely to equally affect reproduction – an especially disadvantageous recessive trait expressed in a homozygous recessive individual is likely to eliminate itself, naturally limiting the expression of its phenotype. Third, recessive deleterious alleles will be "masked" by heterozygosity, and so in a dominant-recessive trait, heterozygotes will not be selected against.

When recessive deleterious alleles occur in the heterozygous state, where their potentially deleterious expression is masked by the corresponding wild-type allele, this masking phenomenon is referred to as complementation (see complementation (genetics)).

In general, sexual reproduction in eukaryotes has two fundamental aspects: genetic recombination during meiosis, and outcrossing. It has been proposed that these two aspects have two natural selective advantages respectively. A proposed adaptive advantage of meiosis is that it facilitates recombinational repair of DNA damages that are otherwise difficult to repair (see DNA repair as the adaptive advantage of meiosis). A proposed adaptive advantage of outcrossing is complementation, which is the masking of deleterious recessive alleles^[8][9] (see hybrid vigor or heterosis). The selective advantage of complementation may largely account for the avoidance of inbreeding (see kin recognition).

However animals often do not avoid inbreeding.^[10] Among animals, inbreeding avoidance is highly variable.^[11] Inbreeding avoidance through mate selection appears to evolve only when there is both a risk of inbreeding depression and there also are frequent encounters between potential sexual partners that are related to each other.^[11]

Hybridization as a conservation effort is appropriate if the population has lost "substantial genetic variation through genetic drift and the detrimental effects of inbreeding depression are apparent" and a similar population should be used.^[12][13] Different populations of the same species have different deleterious traits, and therefore their cross breeding is less likely to result in homozygosity at most loci in the offspring. This is known as outbreeding enhancement, which can be performed in extreme cases of severe inbreeding^[12] by conservation managers and zoo captive breeders to prevent inbreeding depression.

However, intermixing two different populations can give rise to unfit polygenic traits in outbreeding depression (i.e. yielding offspring which lack the genetic adaptations to specific environmental conditions). These, then, will have a lowered fitness than pure-bred individuals of a particular subspecies that has adapted to its local environment.

Inbreeding may have both detrimental and beneficial effects.^[14] The biological effects of inbreeding depression in humans can on occasion be confounded by socioeconomic and cultural influences on reproductive behavior.^[15] Studies in human populations have shown that age at marriage, duration of marriage, illiteracy, contraceptive use, and reproductive compensation are the major determinants of apparent fertility, even amongst populations with a high proportion of consanguinous unions.^[16] However, several small effects on increased mortality,^[17] longer inter-birth intervals^[17] and reduced overall productivity^[15] have been noted in certain isolated populations, though another study about the Icelandic population suggests no effect on lifespan past the 2nd cousin level.^[18]

Charles Darwin was one of the first scientists to demonstrate the effects of inbreeding depression, through numerous experiments on plants. Darwin's wife, Emma, was his first cousin, and he was concerned about the impact of inbreeding on his ten children, three of whom died at age ten or younger; three others had childless long-term marriages.^[19][20][21]

Humans do not seek to completely minimize inbreeding, but rather to maintain an optimal amount of inbreeding vs. outbreeding. Close inbreeding reduces fitness through inbreeding depression, but some inbreeding brings benefits.^[22][23] Indeed, inbreeding "increases the speed of selection of beneficial recessive and co-dominant alleles, e.g. those that protect against diseases."^[24] In general mating between humans who have an equivalent relatedness closer to that of third cousins results in reduced fitness in the children. By contrast outbreeding in humans at worst results in certain autoimmune conditions e.g. asthma.^{[citation needed]}

A small isolated highly inbred population of gray wolves in Isle Royale National Park, Michigan, USA, was considered in 2019 to be at imminent risk of extinction.^[25] This gray wolf population had been experiencing severe inbreeding depression primarily due to the homozygous expression of strongly deleterious recessive mutations.^[25][26] Defects arising from severe inbreeding among the wolves included reduced survival and reproduction, malformed vertebrae, syndactyly, probable cataracts, an unusual "rope tail" and anomalous fur phenotypes.^[25] A separate small inbred population of gray wolves in Scandinavia was also found to suffer from inbreeding depression due to the homozygous expression of deleterious recessive mutations.^[27]

Whilst inbreeding depression has been found to occur in almost all sufficiently studied species, some taxa, most notably some angiosperms, appear to suffer lower fitness costs than others in inbred populations.^[28] Three mechanisms appear to be responsible for this: purging, differences in ploidy, and selection for heterozygosity.^[28] It must be cautioned that some studies failing to show an absence of inbreeding depression in certain species can arise from small sample sizes or where the supposedly outbred control group is already suffering inbreeding depression, which frequently occurs in populations that have undergone a recent bottleneck, such as those of the naked mole rat.^[28][29]

Purging selection occurs where the phenotypes of deleterious recessive alleles are exposed through inbreeding, and thus can be selected against. This can lead to such detrimental mutations being removed from the population, and has been demonstrated to occur rapidly where the recessive alleles have a lethal effect.^[28] The efficiency of purging will depend on the relationship between the magnitude of the deleterious effect that is unmasked in the homozygotes and the importance of genetic drift, so that purging is weaker for non-lethal than for recessive lethal alleles.^[30] For very small populations, drift has a strong influence, which can cause the fixation of sublethal alleles under weak selection.^[28] The fixation of a single allele for a specific gene can also reduce fitness where heterozygote advantage was previously present (i.e., where heterozygous individuals have higher fitness than homozygotes of either allele), although this phenomenon seems to make a usually small contribution to inbreeding depression. Although naturally occurring, purging can be important for population survival, deliberately attempting to purge deleterious mutations from a population is not generally recommended as a technique to improve the fitness of captive bred animals.^[31][32][33] In plants, genetic load can be assessed through a test analogous to an inbreeding depression test called an Autogamy depression test.

Many angiosperms (flowering plants) can self-fertilise for several generations and suffer little from inbreeding depression. This is very useful for species which disperse widely and can therefore find themselves growing in a novel environment with no conspecifics present.^[28] Polyploidy (having more than two paired sets of each chromosome), which is prevalent in angiosperms, ferns and a select few animal taxa, accounts for this. By having several copies of a chromosome, as opposed to two, homozygosity is less likely to occur in inbred offspring. This means that recessive deleterious alleles are not expressed as frequently as with many copies of a chromosome; it is more likely that at least one will contain a functional allele.^[28]

Selection for heterozygosity is rare, as lost loci undergo purifying selection for homozygous loci.^[4] Inbreeding depression has also been found to occur more gradually than predicted in some wild populations, such as in the highly inbred population of Scandinavian wolves. This appears to be due to a selection pressure for more heterozygous individuals, which generally are in better condition and so are more likely to become one of the few animals to breed and produce offspring.^[34]