Fermat primality test

Fermat's little theorem states that if p is prime and a is not divisible by p, then

${\displaystyle a^{p-1}\equiv 1{\pmod {p}}.}$

If one wants to test whether p is prime, then we can pick random integers a not divisible by p and see whether the congruence holds. If it does not hold for a value of a, then p is composite. This congruence is unlikely to hold for a random a if p is composite.^[1] Therefore, if the equality does hold for one or more values of a, then we say that p is probably prime.

However, note that the above congruence holds trivially for ${\displaystyle a\equiv 1{\pmod {p}}}$, because the congruence relation is compatible with exponentiation. It also holds trivially for ${\displaystyle a\equiv -1{\pmod {p}}}$ if p is odd, for the same reason. That is why one usually chooses a random a in the interval ${\displaystyle 1<a<p-1}$.

Any a such that

${\displaystyle a^{n-1}\equiv 1{\pmod {n}}}$

when n is composite is known as a Fermat liar. In this case n is called Fermat pseudoprime to base a.

If we do pick an a such that

${\displaystyle a^{n-1}\not \equiv 1{\pmod {n}}}$

then a is known as a Fermat witness for the compositeness of n.

Suppose we wish to determine whether n = 221 is prime. Randomly pick 1 < a < 220, say a = 38. We check the above congruence and find that it holds:

${\displaystyle a^{n-1}=38^{220}\equiv 1{\pmod {221}}.}$

Either 221 is prime, or 38 is a Fermat liar, so we take another a, say 24:

${\displaystyle a^{n-1}=24^{220}\equiv 81\not \equiv 1{\pmod {221}}.}$

So 221 is composite and 38 was indeed a Fermat liar. Furthermore, 24 is a Fermat witness for the compositeness of 221.

The algorithm can be written as follows:

Inputs: n: a value to test for primality, n>3; k: a parameter that determines the number of times to test for primality

Output: composite if n is composite, otherwise probably prime

Repeat k times:

Pick a randomly in the range [2, n − 2]

If ${\displaystyle a^{n-1}\not \equiv 1{\pmod {n}}}$, then return composite

If composite is never returned: return probably prime

The a values 1 and n − 1 are not used as the equality holds for all n and all odd n respectively, hence testing them adds no value.

Using fast algorithms for modular exponentiation and multiprecision multiplication, the running time of this algorithm is O(k log²n log log n) = Õ(k log²n), where k is the number of times we test a random a, and n is the value we want to test for primality; see Miller–Rabin primality test for details.

There are infinitely many Fermat pseudoprimes to any given basis a > 1.^{[1]: Theorem 1} Even worse, there are infinitely many Carmichael numbers.^[2] These are numbers ${\displaystyle n}$ for which all values of ${\displaystyle a}$ with ${\displaystyle \operatorname {gcd} (a,n)=1}$ are Fermat liars. For these numbers, repeated application of the Fermat primality test performs the same as a simple random search for factors. While Carmichael numbers are substantially rarer than prime numbers (Erdős' upper bound for the number of Carmichael numbers^[3] is lower than the prime number function n/log(n)) there are enough of them that Fermat's primality test is not often used in the above form. Instead, other more powerful extensions of the Fermat test, such as Baillie–PSW, Miller–Rabin, and Solovay–Strassen are more commonly used.

In general, if ${\displaystyle n}$ is a composite number that is not a Carmichael number, then at least half of all

${\displaystyle a\in (\mathbb {Z} /n\mathbb {Z} )^{*}}$ (i.e. ${\displaystyle \operatorname {gcd} (a,n)=1}$)

are Fermat witnesses. For proof of this, let ${\displaystyle a}$ be a Fermat witness and ${\displaystyle a_{1}}$, ${\displaystyle a_{2}}$, ..., ${\displaystyle a_{s}}$ be Fermat liars. Then

${\displaystyle (a\cdot a_{i})^{n-1}\equiv a^{n-1}\cdot a_{i}^{n-1}\equiv a^{n-1}\not \equiv 1{\pmod {n}}}$

and so all ${\displaystyle a\cdot a_{i}}$ for ${\displaystyle i=1,2,...,s}$ are Fermat witnesses.

Analogous to the Lucas–Lehmer residue, ${\displaystyle r=a^{n-1}{\pmod {n}}}$ is called the Fermet residue of n to base a. There are a few variants that produce different types of residues,^[4] most importantly the strong probable prime (SPRP) residue.^[5]

If the residue r for n to base a is known, then for any proper divisor k of n, it is possible to perform a quick though weaker primality test on n/k. If n/k is prime, by Fermat's theorem ${\textstyle a^{n/k}=a^{k}{\pmod {n/k}}}$ and ${\textstyle a^{n}=a^{k}{\pmod {n/k}}}$. As a result ${\textstyle r=a^{k-1}{\pmod {n/k}}}$, which can be much more efficiently checked for values of k much smaller than n. (This is the method used by the Great Internet Mersenne Prime Search for testing cofactors.)^[4]

A even weaker form of the test can be conducted with a truncated ${\displaystyle a^{n}{\pmod {n}}{\pmod {2^{t}}}}$ if storage space for the residue is a concern, so long as ${\textstyle d\leq 2^{t}}$ and ${\displaystyle n>2^{t}}$. Let ${\textstyle s=a^{k}{\pmod {n/k}}}$, then ${\textstyle r=s+wn/k}$ for some w. Take this mod 2^t and we have ${\textstyle w=(n/k)^{-1}(s-r){\pmod {2^{t}}}}$. n/k is composite if ${\displaystyle w\geq d}$. A similar test can be done on truncated Lucas–Lehmer residues.^[6]

It is also true that if all bases a are systematically checked in the interval ${\displaystyle [2,{\sqrt {n}}]}$, each demonstrating congruence to 1, the test is effectively deterministic. We may say that n is definitely prime. At cursory glance one may assume that n exists in the union of both primes and Carmichael numbers for such a scenario, but if a-values are systematically checked in the interval, one is bound to be a prime factor of a composite n at some point before ${\displaystyle {\sqrt {n}}}$, thus making a and n not coprime, and failing the congruence, even if n is indeed a Carmichael number. Carmichael numbers do fail Fermat's congruence to 1 if the base used is not coprime. Though this is more computationally expensive than brute force divisibility checking (trial division), it is of theoretical value.

As mentioned above, most applications use a Miller–Rabin or Baillie–PSW test for primality. Sometimes a Fermat test (along with some trial division by small primes) is performed first to improve performance. GMP since version 3.0 uses a base-210 Fermat test after trial division and before running Miller–Rabin tests. Libgcrypt uses a similar process with base 2 for the Fermat test, but OpenSSL does not.

In practice with most big number libraries such as GMP, the Fermat test is not noticeably faster than a Miller–Rabin test, and can be slower for many inputs.^[7]

As an exception, OpenPFGW uses only the Fermat test for probable prime testing. The program is typically used with multi-thousand digit inputs with a goal of maximum speed with very large inputs. Another well known program that relies only on the Fermat test is PGP where it is only used for testing of self-generated large random values (an open source counterpart, GNU Privacy Guard, uses a Fermat pretest followed by Miller–Rabin tests).

Internet volunteer computing projects such as Great Internet Mersenne Prime Search (GIMPS) and PrimeGrid use the Fermat primality test because there is an efficient proof scheme (Gerbicz-Li) for modular exponentiation. Selected intermediate results, combined with a verifiable delay function, are used to generate a "proof" file for verifying the authenticity and correctness of the computation, protecting against both hardware error and malicious actors. This proof is hard to forge given a low order assumption. The original form of the verification (Gerbicz-Pietrzak) only worked with n being derivable from powers of 2, such as in the case of Mersenne primes, Mersenne cofactors, and Proth primes; Li's modification generalizes it to any n.^[8]

The GIMPS in particular tests Mersenne primes and Mersenne cofactors. The default is to use a = 3 as all Mersenne numbers would pass the test with a = 2.