Hubbry Logo
Frequency analysisFrequency analysisMain
Open search
Frequency analysis
Community hub
Frequency analysis
logo
8 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Frequency analysis
Frequency analysis
Frequency analysis
View on Wikipedia
from Wikipedia
A typical distribution of letters in English language text. Weak ciphers do not sufficiently mask the distribution, and this might be exploited by a cryptanalyst to read the message.

In cryptanalysis, frequency analysis (also known as counting letters) is the study of the frequency of letters or groups of letters in a ciphertext. The method is used as an aid to breaking classical ciphers.

Frequency analysis is based on the fact that, in any given stretch of written language, certain letters and combinations of letters occur with varying frequencies. Moreover, there is a characteristic distribution of letters that is roughly the same for almost all samples of that language. For instance, given a section of English language, E, T, A and O are the most common, while Z, Q, X and J are rare. Likewise, TH, ER, ON, and AN are the most common pairs of letters (termed bigrams or digraphs), and SS, EE, TT, and FF are the most common repeats.[1] The nonsense phrase etaoin shrdlu represents the 12 most frequent letters in typical English language text.

In some ciphers, such properties of the natural language plaintext are preserved in the ciphertext, and these patterns have the potential to be exploited in a ciphertext-only attack.

Frequency analysis for simple substitution ciphers

[edit]

In a simple substitution cipher, each letter of the plaintext is replaced with another, and any particular letter in the plaintext will always be transformed into the same letter in the ciphertext. For instance, if all occurrences of the letter e turn into the letter X, a ciphertext message containing numerous instances of the letter X would suggest to a cryptanalyst that X represents e.

The basic use of frequency analysis is to first count the frequency of ciphertext letters and then associate guessed plaintext letters with them. More Xs in the ciphertext than anything else suggests that X corresponds to e in the plaintext, but this is not certain; t and a are also very common in English, so X might be either of them. It is unlikely to be a plaintext z or q, which are less common. Thus the cryptanalyst may need to try several combinations of mappings between ciphertext and plaintext letters.

More complex use of statistics can be conceived, such as considering counts of pairs of letters (bigrams), triplets (trigrams), and so on. This is done to provide more information to the cryptanalyst, for instance, Q and U nearly always occur together in that order in English, even though Q itself is rare.

An example

[edit]

Suppose Eve has intercepted the cryptogram below, and it is known to be encrypted using a simple substitution cipher: 

LIVITCSWPIYVEWHEVSRIQMXLEYVEOIEWHRXEXIPFEMVEWHKVSTYLXZIXLIKIIXPIJVSZEYPERRGERIM
WQLMGLMXQERIWGPSRIHMXQEREKIETXMJTPRGEVEKEITREWHEXXLEXXMZITWAWSQWXSWEXTVEPMRXRSJ
GSTVRIEYVIEXCVMUIMWERGMIWXMJMGCSMWXSJOMIQXLIVIQIVIXQSVSTWHKPEGARCSXRWIEVSWIIBXV
IZMXFSJXLIKEGAEWHEPSWYSWIWIEVXLISXLIVXLIRGEPIRQIVIIBGIIHMWYPFLEVHEWHYPSRRFQMXLE
PPXLIECCIEVEWGISJKTVWMRLIHYSPHXLIQIMYLXSJXLIMWRIGXQEROIVFVIZEVAEKPIEWHXEAMWYEPP
XLMWYRMWXSGSWRMHIVEXMSWMGSTPHLEVHPFKPEZINTCMXIVJSVLMRSCMWMSWVIRCIGXMWYMX

For this example, uppercase letters are used to denote ciphertext, lowercase letters are used to denote plaintext (or guesses at such), and X~t is used to express a guess that ciphertext letter X represents the plaintext letter t.

Eve could use frequency analysis to help solve the message along the following lines: counts of the letters in the cryptogram show that I is the most common single letter,[2] XL most common bigram, and XLI is the most common trigram. e is the most common letter in the English language, th is the most common bigram, and the is the most common trigram. This strongly suggests that X~t, L~h and I~e. The second most common letter in the cryptogram is E; since the first and second most frequent letters in the English language, e and t are accounted for, Eve guesses that E~a, the third most frequent letter. Tentatively making these assumptions, the following partial decrypted message is obtained. 

heVeTCSWPeYVaWHaVSReQMthaYVaOeaWHRtatePFaMVaWHKVSTYhtZetheKeetPeJVSZaYPaRRGaReM
WQhMGhMtQaReWGPSReHMtQaRaKeaTtMJTPRGaVaKaeTRaWHatthattMZeTWAWSQWtSWatTVaPMRtRSJ
GSTVReaYVeatCVMUeMWaRGMeWtMJMGCSMWtSJOMeQtheVeQeVetQSVSTWHKPaGARCStRWeaVSWeeBtV
eZMtFSJtheKaGAaWHaPSWYSWeWeaVtheStheVtheRGaPeRQeVeeBGeeHMWYPFhaVHaWHYPSRRFQMtha
PPtheaCCeaVaWGeSJKTVWMRheHYSPHtheQeMYhtSJtheMWReGtQaROeVFVeZaVAaKPeaWHtaAMWYaPP
thMWYRMWtSGSWRMHeVatMSWMGSTPHhaVHPFKPaZeNTCMteVJSVhMRSCMWMSWVeRCeGtMWYMt

Using these initial guesses, Eve can spot patterns that confirm her choices, such as "that". Moreover, other patterns suggest further guesses. "Rtate" might be "state", which would mean R~s. Similarly "atthattMZe" could be guessed as "atthattime", yielding M~i and Z~m. Furthermore, "heVe" might be "here", giving V~r. Filling in these guesses, Eve gets: 

hereTCSWPeYraWHarSseQithaYraOeaWHstatePFairaWHKrSTYhtmetheKeetPeJrSmaYPassGasei
WQhiGhitQaseWGPSseHitQasaKeaTtiJTPsGaraKaeTsaWHatthattimeTWAWSQWtSWatTraPistsSJ
GSTrseaYreatCriUeiWasGieWtiJiGCSiWtSJOieQthereQeretQSrSTWHKPaGAsCStsWearSWeeBtr
emitFSJtheKaGAaWHaPSWYSWeWeartheStherthesGaPesQereeBGeeHiWYPFharHaWHYPSssFQitha
PPtheaCCearaWGeSJKTrWisheHYSPHtheQeiYhtSJtheiWseGtQasOerFremarAaKPeaWHtaAiWYaPP
thiWYsiWtSGSWsiHeratiSWiGSTPHharHPFKPameNTCiterJSrhisSCiWiSWresCeGtiWYit

In turn, these guesses suggest still others (for example, "remarA" could be "remark", implying A~k) and so on, and it is relatively straightforward to deduce the rest of the letters, eventually yielding the plaintext. 

hereuponlegrandarosewithagraveandstatelyairandbroughtmethebeetlefromaglasscasei
nwhichitwasencloseditwasabeautifulscarabaeusandatthattimeunknowntonaturalistsof
courseagreatprizeinascientificpointofviewthereweretworoundblackspotsnearoneextr
emityofthebackandalongoneneartheotherthescaleswereexceedinglyhardandglossywitha
lltheappearanceofburnishedgoldtheweightoftheinsectwasveryremarkableandtakingall
thingsintoconsiderationicouldhardlyblamejupiterforhisopinionrespectingit

At this point, it would be a good idea for Eve to insert spaces and punctuation: 

Hereupon Legrand arose, with a grave and stately air, and brought me the beetle
from a glass case in which it was enclosed. It was a beautiful scarabaeus, and, at
that time, unknown to naturalists—of course a great prize in a scientific point
of view. There were two round black spots near one extremity of the back, and a
long one near the other. The scales were exceedingly hard and glossy, with all the
appearance of burnished gold. The weight of the insect was very remarkable, and,
taking all things into consideration, I could hardly blame Jupiter for his opinion
respecting it.

In this example from "The Gold-Bug", Eve's guesses were all correct. This would not always be the case, however; the variation in statistics for individual plaintexts can mean that initial guesses are incorrect. It may be necessary to backtrack incorrect guesses or to analyze the available statistics in much more depth than the somewhat simplified justifications given in the above example.

It is possible that the plaintext does not exhibit the expected distribution of letter frequencies. Shorter messages are likely to show more variation. It is also possible to construct artificially skewed texts. For example, entire novels have been written that omit the letter e altogether — a form of literature known as a lipogram.

History and usage

[edit]
First page of Al-Kindi's 9th century Manuscript on Deciphering Cryptographic Messages
Arabic Letter Frequency distribution

The first known recorded explanation of frequency analysis (indeed, of any kind of cryptanalysis) was given in the 9th century by Al-Kindi, an Arab polymath, in A Manuscript on Deciphering Cryptographic Messages.[3] It has been suggested that a close textual study of the Qur'an first brought to light that Arabic has a characteristic letter frequency.[4] Its use spread, and similar systems were widely used in European states by the time of the Renaissance. By 1474, Cicco Simonetta had written a manual on deciphering encryptions of Latin and Italian text.[5]

Several schemes were invented by cryptographers to defeat this weakness in simple substitution encryptions. These included:

  • Homophonic substitution: Use of homophones — several alternatives to the most common letters in otherwise monoalphabetic substitution ciphers. For example, for English, both X and Y ciphertext might mean plaintext E.
  • Polyalphabetic substitution, that is, the use of several alphabets — chosen in assorted, more or less devious, ways (Leone Alberti seems to have been the first to propose this); and
  • Polygraphic substitution, schemes where pairs or triplets of plaintext letters are treated as units for substitution, rather than single letters, for example, the Playfair cipher invented by Charles Wheatstone in the mid-19th century.

A disadvantage of all these attempts to defeat frequency counting attacks is that it increases complication of both enciphering and deciphering, leading to mistakes. Famously, a British Foreign Secretary is said to have rejected the Playfair cipher because, even if school boys could cope successfully as Wheatstone and Playfair had shown, "our attachés could never learn it!".

The rotor machines of the first half of the 20th century (for example, the Enigma machine) were essentially immune to straightforward frequency analysis. However, other kinds of analysis ("attacks") successfully decoded messages from some of those machines.[6]

Letter frequency in Spanish

Frequency analysis requires only a basic understanding of the statistics of the plaintext language and some problem-solving skills, and, if performed by hand, tolerance for extensive letter bookkeeping. During World War II, both the British and the Americans recruited codebreakers by placing crossword puzzles in major newspapers and running contests for who could solve them the fastest. Several of the ciphers used by the Axis powers were breakable using frequency analysis, for example, some of the consular ciphers used by the Japanese. Mechanical methods of letter counting and statistical analysis (generally IBM card type machinery) were first used in World War II, possibly by the US Army's SIS. Today, the work of letter counting and analysis is done by computer software, which can carry out such analysis in seconds. With modern computing power, classical ciphers are unlikely to provide any real protection for confidential data.

Frequency analysis in fiction

[edit]
Part of the cryptogram in The Dancing Men

Frequency analysis has been described in fiction. Edgar Allan Poe's "The Gold-Bug" and Sir Arthur Conan Doyle's Sherlock Holmes tale "The Adventure of the Dancing Men" are examples of stories which describe the use of frequency analysis to attack simple substitution ciphers. The cipher in the Poe story is encrusted with several deception measures, but this is more a literary device than anything significant cryptographically.

See also

[edit]

Further reading

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Frequency analysis

View on Grokipedia
from Grokipedia
Frequency analysis is a fundamental technique in that involves studying the frequency of occurrence of letters, symbols, or groups thereof in a to infer the underlying , particularly effective against monoalphabetic substitution ciphers such as the . This method exploits the predictable patterns in natural languages, where certain letters like '', '', 'A', and '' appear far more frequently in English texts than rarer ones such as '', '', or '', allowing cryptanalysts to map symbols to their plaintext equivalents by comparing frequency distributions. The origins of frequency analysis trace back to the , when the Arab polymath (c. 801–873 CE) developed it systematically in his treatise A Manuscript on Deciphering Cryptographic Messages, marking the first known recorded explanation of any cryptanalytic technique. 's innovation involved tallying letter frequencies in both known samples and encrypted texts, then aligning the most common symbols in the with the most frequent letters in the target language to partially or fully decrypt messages, a process that relied on early statistical insights derived from linguistic analysis. This breakthrough not only weakened simple substitution ciphers but also spurred advancements in , as encryphers sought more complex methods like polyalphabetic substitution to evade detection. In practice, frequency analysis begins with collecting a sufficiently long —ideally hundreds of characters—to ensure reliable statistics, followed by ranking symbols by occurrence and hypothesizing mappings based on language norms; for instance, the most frequent ciphertext letter might correspond to 'E' in English, with trial substitutions revealing patterns like common words or digrams (e.g., 'TH' or 'HE'). While highly effective against classical ciphers, its utility diminishes against modern polyalphabetic or computationally secure systems, though it remains a educational tool in understanding cryptographic vulnerabilities and has influenced fields beyond cryptology, including and .

Fundamentals

Definition and Basic Principles

Frequency analysis is a cryptographic technique that involves counting and comparing the relative frequencies of symbols, letters, or other units within a text or data stream to reveal underlying patterns or structures. This method exploits the statistical regularities inherent in natural languages and other datasets, where certain elements occur more frequently than others, allowing analysts to infer relationships between ciphertext and plaintext without prior knowledge of the encoding key. At its core, frequency analysis relies on the principle that natural languages exhibit non-uniform distributions of characters, meaning letters do not appear with equal probability. For example, in English, the letters follow an approximate order of frequency remembered by the mnemonic "," where 'e' is the most common, followed by 't', 'a', 'o', 'i', 'n', 's', 'h', 'r', 'd', 'l', and 'u'. This uneven distribution arises from linguistic patterns, such as the prevalence of common words and grammatical structures. In , observed frequencies in an encoded text are compared to these expected frequencies from the source language; significant matches or deviations help identify mappings or anomalies, as substitution ciphers preserve the original frequency profile despite obscuring individual symbols. Mathematically, frequency analysis computes relative frequencies as proportions of occurrences. The relative frequency f(x)f(x) of a symbol xx is given by f(x)=count of xtotal count of all symbols,f(x) = \frac{\text{count of } x}{\text{total count of all symbols}}, yielding values between 0 and 1, often expressed as percentages for interpretation. For instance, the letter 'e' in English text has a relative frequency of approximately 12.7%, making it a key indicator in analysis. This foundational approach enables in encoded texts by highlighting consistencies between anticipated and actual distributions, serving as a prerequisite for more advanced cryptanalytic methods without requiring assumptions about specific encoding schemes.

Frequency Distributions in Natural Language

In , letter frequencies exhibit non-uniform distributions shaped by linguistic structures, with vowels and common consonants appearing far more often than rare ones. These patterns are derived from large corpora of written texts and provide a foundation for analyzing textual regularity. For instance, in English, the letter 'E' occurs approximately 12.02% of the time, followed by 'T' at 9.10% and 'A' at 8.12%, based on a sample of 40,000 words. The following table summarizes the relative frequencies of letters in English, highlighting the dominance of a few characters:
LetterFrequency (%)
E12.02
T9.10
A8.12
O7.68
I7.31
N6.95
S6.28
R6.02
H5.92
D4.32
L3.98
U2.88
C2.71
M2.61
F2.30
Y2.11
W2.09
G2.03
P1.82
B1.49
V1.11
K0.69
Q0.11
X0.17
J0.10
Z0.07
Digraph frequencies further reveal pairwise patterns, with common combinations like "TH" at 1.52%, "HE" at 1.28%, "IN" at 0.94%, and "ER" at 0.94% in English texts from the same corpus. Similar distributions appear in other major languages using the Latin alphabet, though rankings vary due to phonological differences. In French, 'E' leads at 15.10%, followed by 'A' at 8.13%, 'S' at 7.91%, 'T' at 7.11%, and 'I' at 6.94%; in Spanish, 'E' is 13.72%, 'A' 11.72%, 'O' 8.44%, 'S' 7.20%, and 'N' 6.83%; while in German, 'E' tops at 16.93%, 'N' 10.53%, 'I' 8.02%, 'R' 6.89%, and 'S' 6.42%. These values are derived from large text corpora. Phonetic factors, such as the prevalence of vowels in structures, contribute to higher frequencies for letters representing them (e.g., , A, O across languages), while orthographic conventions like silent letters or digraphs for sounds alter distributions. Cultural influences, including loanwords from other languages and historical spelling reforms, also shift frequencies; for example, increased use of borrowed terms can elevate certain consonants in modern texts. A key quantitative measure of these distributions is the index of coincidence (IC), defined as IC=∑i=126fi2IC = \sum_{i=1}^{26} f_i^2, where fif_i is the relative frequency of the i-th letter, which quantifies deviation from uniformity. For English, IC ≈ 0.066, compared to ≈ 0.038 for random text over 26 symbols, reflecting the redundancy inherent in natural language. Frequencies vary by dialect (e.g., shows slightly higher 'U' usage than American due to spellings like "colour"), genre (formal favors longer words with more vowels, while informal text increases contractions and ), and sample length (short texts exhibit higher variance, stabilizing in samples over 1,000 characters). These patterns in frequencies serve as a baseline for cryptanalytic tools that detect deviations in encrypted texts.

Cryptanalytic Applications

Substitution Ciphers

A monoalphabetic encrypts by replacing each letter with a unique letter according to a fixed , thereby preserving the relative distribution of letters from the original . This preservation occurs because the substitution is a one-to-one mapping, so the most frequent letters remain the most frequent in , albeit under different symbols. To break such a using frequency analysis, the cryptanalyst first tallies the frequencies of letters in the and compares them to known distributions, such as English where 'E' appears approximately 12.7% of the time, followed by 'T' at 9.1%. The most frequent ciphertext letter is then hypothesized to map to 'E', the next to 'T' or 'A', and so on, forming an initial partial key. This mapping is iteratively refined by examining digraphs (pairs of letters) and trigraphs, whose expected frequencies in English—such as 'TH' at about 2.7%—help resolve ambiguities and confirm substitutions. Cryptanalysts employ tools like frequency charts to visualize these distributions and the (IC) to validate mappings, as the IC for a monoalphabetic ciphertext closely matches English's value of around 0.067, indicating non-random repetition patterns. Additionally, the quantifies the goodness-of-fit between observed and expected frequencies in a proposed decryption: χ2=∑(Oi−Ei)2Ei\chi^2 = \sum \frac{(O_i - E_i)^2}{E_i} where OiO_i is the observed count of the ii-th letter in the decrypted text, and EiE_i is the expected count based on language frequencies; lower χ2\chi^2 values suggest a better match to natural language. This method succeeds against monoalphabetic ciphers because the fixed mapping retains detectable frequency patterns, but it fails against polyalphabetic ciphers, which use multiple substitutions to diffuse and flatten letter frequencies, approximating a uniform distribution.

Step-by-Step Example

Consider the short "URFUA FOBRF MOBYL KFRBF KXDMF XFLBB ZFEUO ZFRKM FEXUO FRKUO LFUAF RBFYA MFURF PMCC", encrypted via a simple where each symbol (including spaces) is replaced by a unique ciphertext letter. Begin by counting the occurrences of each letter to identify patterns matching expected English frequencies, where spaces and letters like E, T, and A appear most often.
Cipher LetterFrequency
F16
R7
U7
B6
M5
O5
A3
L3
X3
C2
E2
K2
Y2
Z2
D1
The highest F (16 occurrences) likely maps to space, a common symbol in English text comprising about 18% of typical messages. Substituting F with a space reveals word boundaries: UR UA OBR MOBYLK RB KXDM X FLBBZ EUOZ RKM EXUO RKUOL UA RB YA M UR PMCC. Next, rank the remaining letters by frequency and hypothesize mappings to English letters (E ≈12.7%, T ≈9.1%, A ≈8.2%). R and U (7 each) are candidates for E or T. Trial mapping R to T (common initial digraph in words like "THE") and U to I (fitting short words like "IT" for UR) yields partial decryption: IT IS NOT ENO?G? TO H??? A ??BB? MIO? T?M M?IO T?IOG IS TO ?A E IT ?MCC. This produces recognizable fragments like "IT IS NOT" and "TO". Refine by incorporating digraph frequencies; for instance, RB (appearing twice) maps to TO with B to O, updating to: IT IS NOT ENO?G? TO HA?E A GOOD MION T?E E?IO T?IOG IS TO ?A E IT ?MCC. Continuing iteratively, MOBYLK suggests "ENOUGH" (M to E, O to N, Y to U, L to G, K to H), and RKM to "THE" (confirming R to T, K to H, M to E). Further trials adjust X to A (KXDM to "HAVE"), Z to D (FLBBZ to "GOOD"), E to M (EUOZ to "MIND"), and so on, resolving ambiguities through . The evolving mapping table illustrates progress: Initial Mapping
CipherPlain
F(space)
RT
UI
Intermediate Mapping (after digraphs)
CipherPlain
F(space)
RT
UI
BO
ME
KH
ON
YU
LG
Final Mapping
CipherPlain
F(space)
RT
UI
BO
ME
KH
ON
AS
YU
LG
XA
ZD
EM
CL
DV
PW
Applying the complete mapping decrypts the text to: "IT IS NOT ENOUGH TO HAVE A GOOD MIND THE MAIN THING IS TO USE IT WELL." This process highlights the role of in identifying likely mappings and the iterative trial-and-error nature of frequency analysis, where initial guesses are refined based on emerging readable words and n-grams like "THE" or "TO." An optional can validate mappings by comparing observed digram frequencies to English expectations, though manual iteration often suffices for short texts.

Advanced Techniques and Limitations

While basic frequency analysis excels against monoalphabetic substitution ciphers, extensions enable its application to more complex polyalphabetic systems. The , developed by Friedrich Kasiski in 1863, attacks ciphers like the Vigenère by identifying repeated strings of three or more characters in the and calculating the distances between their occurrences; these distances are often multiples of the key length, allowing estimation via their . Complementing this, the —introduced by William Friedman in the 1920s—can be computed on sliding windows of the to detect periodicity, as windows aligned with the key length exhibit higher values akin to monoalphabetic text (approximately 0.065 for English), while misaligned windows approach random uniformity (0.038). For enhanced precision in substitution , and analysis builds on unigram frequencies by examining pairwise or triple character patterns, revealing contextual redundancies like common English digraphs ("th," "he") that single-letter counts overlook. Despite these advances, frequency analysis suffers from inherent limitations that reduce its reliability in certain scenarios. It performs poorly on short texts under 100 letters, as the sample size yields unreliable frequency estimates lacking sufficient statistical power to match against known distributions. For instance, short Caesar ciphertexts with only 7 letters lack sufficient data for reliable determination of letter frequencies. Homophonic substitution ciphers counter this by employing one-to-many mappings, where frequent plaintext letters (e.g., 'e') are represented by multiple ciphertext symbols, equalizing overall frequencies and obscuring high-probability matches. The technique also fails against non- data, such as random binary streams or encoded numbers, which lack the predictable letter distributions of natural languages. Additionally, deliberate insertion of padding or nulls—meaningless filler symbols like 'x'—disrupts counts by artificially inflating less common letters or altering expected patterns at message ends. Cipher designers have developed countermeasures to mitigate these vulnerabilities and flatten frequency profiles. Keyword-based substitutions, as in the , cycle through multiple alphabets derived from a repeating keyword, distributing letter frequencies across positions and thwarting direct matching. Transposition ciphers rearrange positions without changing letter frequencies, preserving language-like distributions that identify the cipher type but complicating key recovery by scrambling sequential patterns needed for analysis. Modern padding schemes, including homophonic encoding, further equalize distributions by assigning multiple representations to elements proportional to their natural frequencies, rendering statistically uniform. In modern contexts, computational implementations of frequency analysis enhance brute-force of classical ciphers through automated tools that integrate n-gram counts, Kasiski tests, and calculations for rapid key space reduction.

Historical Context

Origins and Early Methods

The origins of frequency analysis trace back to the 9th century in the Islamic world, where it emerged as a systematic method for deciphering substitution ciphers. , an Arab also known as Alkindus, is credited with developing the foundational technique in his treatise Risala fi fī istikhrāj al-muʿamma (A Manuscript on Deciphering Cryptographic Messages), written around 830 CE. The manuscript was lost for most of history and rediscovered in the Süleymaniye Library in in the late 20th century, with its contents published in 2003. In this work, he introduced the concept of counting the frequency of letters in and comparing them to known frequencies in the language, particularly drawing from patterns observed in the , to identify likely substitutions. This approach marked the first known use of statistical analysis in cryptology, enabling the breaking of monoalphabetic ciphers used for diplomatic and military secrets. In medieval , frequency analysis began to appear in rudimentary forms during the , primarily in response to the growing use of ciphers in Italian diplomacy. Amid the fragmented city-states of Renaissance , such as and , basic tallying methods were employed to analyze letter frequencies in intercepted messages, often as part of efforts. These early European techniques involved manual counts of symbols in to match against Latin or vernacular letter distributions, though they remained and less formalized than Al-Kindi's method. A key milestone in this evolution occurred in 1467 with Leon Battista Alberti's De Cifris, a that acknowledged the vulnerability of simple substitution ciphers to frequency-based attacks. Alberti, an humanist and architect, described how frequent letters like vowels could be identified through counting, but he did not elaborate on a full attack methodology; instead, he proposed polyalphabetic ciphers to obscure such patterns and render frequency analysis ineffective. This reference highlighted an emerging awareness of statistical weaknesses in encryption, though practical application in Europe lagged behind conceptual recognition. The initial interest in pattern-breaking through frequency analysis was driven by the exigencies of trade, warfare, and scholarship in interconnected Mediterranean societies. In the Islamic caliphates, expanding trade networks and military campaigns necessitated secure communications, prompting innovations like Al-Kindi's to protect state secrets. Similarly, in 15th-century , intense rivalries among city-states fueled diplomatic intrigue and , where breaking enemy codes could yield strategic advantages in alliances or conflicts. Scholarly pursuits, including the translation of Arabic scientific texts into Latin, facilitated the cross-cultural transmission of cryptanalytic ideas, embedding frequency analysis within broader intellectual efforts to decode ancient and foreign writings.

Key Developments and Practitioners

In the 19th century, frequency analysis advanced significantly through the efforts of , who around 1846 independently broke the Vigenère polyalphabetic cipher by identifying repeated sequences to determine the key length, enabling frequency analysis on the individual substitution alphabets, though he never published his method in detail. Building on such insights, Friedrich Kasiski formalized a systematic approach in his 1863 book Die Geheimschriften und die Dechiffrirkunst, introducing the to determine the periodicity of repeating keywords in s by measuring distances between repeated letter sequences in ciphertext, enabling subsequent frequency analysis on aligned segments. Decades earlier, bridged theoretical and public interest by popularizing frequency-based decryption in his 1841 essay "A Few Words on Secret Writing" and his 1843 "," where the protagonist solves a through distributions, inspiring widespread amateur engagement with the technique. Entering the early 20th century, William Friedman refined frequency analysis for polyalphabetic systems by developing the in the 1920s, a statistical measure quantifying the probability of repeated letters in to estimate key length more reliably than visual frequency inspection alone. Collaborating in the U.S. cryptologic community, advanced statistical through her manual breakdowns of Japanese diplomatic codes like the Red and Blue systems in the 1920s and 1930s, applying frequency patterns and numeral distributions to unravel superencipherments, while training generations of analysts in these methods. During , frequency analysis played a limited role in attacking the due to its rotor design flattening letter distributions, but initial efforts by Polish cryptanalysts in relied on mathematical models, including analysis and exploitation of message indicators from captured documents, to infer rotor wirings. Post-war, the advent of computers transformed frequency analysis from labor-intensive manual tabulation to automated processing, allowing rapid computation of letter distributions and indices on vast ciphertexts, as seen in early U.S. systems that integrated electronic aids for statistical . Historian David Kahn's 1967 comprehensively documented these evolutions, drawing on declassified archives to trace frequency analysis from its precursors—like Al-Kindi's 9th-century foundations—to its mechanized modern forms.

Broader Applications

Linguistics and Text Analysis

In linguistics, frequency analysis plays a crucial role in examining the structure of through large corpora, particularly in and morphology. By quantifying the occurrence of sounds, syllables, or morphemes, researchers can identify patterns such as allophonic variations or paradigmatic irregularities that deviate from expected distributions. For instance, in , corpus-based frequency counts reveal how often certain phonetic realizations appear in specific contexts, aiding in the modeling of and variation across dialects. In morphology, frequency data helps explain productivity and complexity; high-frequency affixes tend to be more regular and less phonologically conditioned, while low-frequency ones exhibit greater irregularity. A foundational principle here is , which posits that word frequency f(r)f(r) is inversely proportional to its rank rr in a corpus, i.e., f(r)∝1rf(r) \propto \frac{1}{r}, reflecting efficiency in use and influencing morphological simplification. Stylometry, a subfield leveraging frequency profiles, applies these methods to attribute authorship by comparing rates of function words, sentence lengths, or lexical choices across texts. Pioneering work analyzed the disputed Federalist Papers (1787–1788), a collection of 85 essays promoting the , where 12 were unattributed among , , and . Using multivariate analysis of word frequencies—such as "upon" and "whilst"—Mosteller and Wallace determined Madison as the likely author of all disputed papers, with posterior probabilities exceeding 0.95 for most, establishing 's forensic reliability. This approach has since informed literary and historical attributions, emphasizing stable stylistic markers over content. Tools like AntConc facilitate such analyses by enabling concordancing and n-gram frequency extraction from corpora, allowing users to generate keyword lists and profiles efficiently. In forensics, frequency-based detects intrinsically by identifying style shifts within documents, such as anomalous distributions signaling inserted text; classifiers trained on these features achieve detection accuracies above 90% in benchmark corpora. Multilingual frequency analysis supports training by aligning parallel corpora and balancing low-resource languages through rare n-grams, improving model robustness; for example, adjusting training data proportions based on token frequencies enhances zero-shot performance across 100+ languages.

Signal Processing and Statistics

In , frequency analysis plays a crucial role in decomposing signals into their constituent frequency components, enabling the identification of underlying patterns and facilitating targeted manipulations. The (DFT) is a fundamental technique for this purpose, converting a finite sequence of equally spaced samples of a time-domain signal into a sequence of frequency-domain coefficients. The DFT is mathematically defined as X(k)=∑n=0N−1x(n)e−j2πkn/N,X(k) = \sum_{n=0}^{N-1} x(n) e^{-j 2\pi k n / N}, where x(n)x(n) represents the input signal samples for n=0n = 0 to N−1N-1, and kk indexes the frequency bins from 0 to N−1N-1. This transform reveals the spectral content of the signal, allowing engineers to isolate specific frequencies for processing. In audio filtering, the DFT is widely applied to remove unwanted noise or enhance particular frequency bands, such as in speech enhancement systems where low-frequency hum is suppressed. Similarly, in vibration analysis, the DFT helps diagnose mechanical faults in machinery by identifying dominant frequencies corresponding to imbalances or bearing defects, as demonstrated in studies of motor vibrations under varying loads. Unlike the discrete symbol counts prevalent in cryptanalysis, frequency analysis in signal processing emphasizes continuous or numerical spectra, where frequencies represent periodic oscillations rather than categorical occurrences. In statistics, frequency analysis shifts to distributional properties of data, using tools like histograms to visualize the empirical frequency distribution of values in a dataset. For categorical data, the probability mass function (PMF) quantifies the likelihood of each category, derived from observed frequencies normalized by the total count, providing a basis for modeling discrete random variables. To assess whether these frequencies conform to an expected uniform or theoretical distribution, the chi-squared goodness-of-fit test is employed, computing the statistic χ2=∑(Oi−Ei)2/Ei\chi^2 = \sum (O_i - E_i)^2 / E_i, where OiO_i and EiE_i are observed and expected frequencies, respectively; significant deviations indicate non-uniformity. Modern applications extend frequency analysis into , particularly for in network traffic, where spectral decomposition via time-frequency methods identifies irregular high-frequency components signaling intrusions or failures. In environments, tools like Hadoop enable scalable frequency counts across massive datasets using distributed paradigms, as seen in word frequency computations on large corpora to uncover patterns without centralized processing bottlenecks. These approaches underscore the versatility of frequency analysis beyond textual domains, focusing on quantitative spectra and to drive insights in and .

Cultural Representations

In Literature and Media

Frequency analysis has been a recurring element in literature and media, often serving as a plot device to showcase intellectual prowess in solving mysteries. Edgar Allan Poe's short story "The Gold-Bug," published in 1843, is widely regarded as the first work of fiction to prominently feature frequency analysis as a method for breaking a substitution cipher. In the narrative, the protagonist William Legrand deciphers a cryptic message leading to buried treasure by counting letter frequencies and mapping them to English patterns, a technique Poe detailed meticulously to engage readers' interest in cryptanalysis. This story not only introduced the term "cryptograph" but also demonstrated the method's accessibility, drawing from Poe's own experiences analyzing reader-submitted ciphers for magazines. The technique appeared again in Arthur Conan Doyle's "The Adventure of the Dancing Men" (1903), where detective applies frequency analysis to decode a series of pictographic symbols representing a threatening a client's safety. Holmes identifies common symbols' occurrences to infer mappings like "E" for the most frequent English letter, unraveling the code step by step. This portrayal influenced later adaptations, including the BBC series Sherlock (2010–2017), where episodes like "" depict Holmes using book ciphers to crack codes, echoing Doyle's original stories involving substitution ciphers and frequency analysis. In film, (2014) alludes to frequency analysis within the context of code-breaking efforts against the , with characters referencing letter distribution analysis for decrypting German messages as a foundational step. Such depictions often employ common tropes, including the archetype of a solitary poring over frequency charts on walls or blackboards to achieve breakthroughs, as seen in Holmes adaptations and films like (2004), where cryptographic solving drives the narrative tension. However, these portrayals frequently include inaccuracies for dramatic effect, such as portraying complex ciphers as solvable in moments through intuitive frequency counts, whereas requires extensive computation and iteration, especially for polyalphabetic systems like Enigma. In The Imitation Game, for instance, the film's compression of historical events oversimplifies the role of frequency methods, blending them with machine-based decryption in ways that prioritize pacing over precision. Media representations have significantly influenced public perception of frequency analysis, popularizing as an intriguing intellectual pursuit and inspiring generations to experiment with codes. Poe's "" in particular sparked widespread amateur interest, leading to a surge in cipher challenges in 19th-century periodicals and laying groundwork for 's cultural allure in . This legacy continues in modern media, fostering educational engagement while sometimes perpetuating myths about the method's simplicity.

References

  1. https://www.101computing.net/frequency-analysis/
  2. https://muslimheritage.com/al-kindi-cryptography/
  3. https://crypto.interactive-maths.com/frequency-analysis-breaking-the-code.html
  4. https://www.telsy.com/en/al-kindi-the-father-of-cryptanalysis/
  5. https://math-sites.uncg.edu/sites/pauli/112/HTML/secfrequency.html
  6. https://www.cs.cornell.edu/~raghu/teaching/Materials_files/hw5.pdf
  7. https://pi.math.cornell.edu/~mec/2003-2004/cryptography/subs/frequencies.html
  8. https://mathcenter.oxford.emory.edu/site/math125/englishLetterFreqs/
  9. https://zoo.cs.yale.edu/classes/cs467/2015f/lectures/ln03.pdf
  10. https://pi.math.cornell.edu/~mec/2003-2004/cryptography/subs/digraphs.html
  11. https://www.sttmedia.com/characterfrequency-french
  12. https://www.sttmedia.com/characterfrequency-spanish
  13. https://www.sttmedia.com/characterfrequency-german
  14. https://www.researchgate.net/publication/324018661_Letter_Frequency_Analysis_of_Languages_Using_Latin_Alphabet
  15. https://www.nsa.gov/portals/75/documents/news-features/declassified-documents/friedman-documents/publications/FOLDER_231/41760429079956.pdf
  16. https://www.math.ucsd.edu/~crypto/Projects/MarshaMoreno/TimeComparisonFrequency.pdf
  17. https://www.sciencedirect.com/topics/computer-science/substitution-cipher
  18. http://practicalcryptography.com/cryptanalysis/letter-frequencies-various-languages/english-letter-frequencies/
  19. https://www.southampton.ac.uk/~wright/1001/chi-squared-test.html
  20. https://www.sciencedirect.com/topics/computer-science/polyalphabetic-cipher
  21. https://people.sc.fsu.edu/~jpeterson/cryptography.pdf
  22. https://web.ecs.syr.edu/courses/cis428/slides/02poly.pdf
  23. https://websites.nku.edu/~christensen/section%2015%20william%20friedman%20index%20of%20coincidence.pdf
  24. http://practicalcryptography.com/cryptanalysis/text-characterisation/monogram-bigram-and-trigram-frequency-counts/
  25. https://www.cs.purdue.edu/homes/ninghui/courses/526_Fall15/handouts/15_526_topic02.pdf
  26. https://www2.rivier.edu/faculty/vriabov/cs572aweb/Assignments/CrackingClassicCiphers.htm
  27. http://www.cs.sjsu.edu/faculty/stamp/RUA/homophonic.pdf
  28. https://eprint.iacr.org/2017/1068.pdf
  29. https://www.infosecinstitute.com/resources/cryptography/cryptanalysis-tools/
  30. https://simonsingh.net/media/articles/maths-and-science/arab-code-breakers/
  31. https://www.cryptool.org/en/education/history/
  32. https://www.cs.trincoll.edu/~crypto/historical/alberti.html
  33. https://www.academia.edu/33813775/Fifteenth_Century_Cryptography_Revisited
  34. https://websites.nku.edu/~christensen/section%2014%20vigenere%20cryptanalysis.pdf
  35. https://www.ciphermachinesandcryptology.com/en/goldbug.htm
  36. https://www.nsa.gov/portals/75/documents/about/cryptologic-heritage/historical-figures-publications/publications/Personalities/the_neglected_giant_agnes_meyer_driscoll.pdf
  37. https://www.nsa.gov/portals/75/documents/about/cryptologic-heritage/historical-figures-publications/publications/wwii/solving_enigma.pdf
  38. https://www.nsa.gov/portals/75/documents/news-features/declassified-documents/cryptologic-histories/cold_war_ii.pdf
  39. https://oxfordre.com/linguistics/display/10.1093/acrefore/9780199384655.001.0001/acrefore-9780199384655-e-145
  40. https://psycnet.apa.org/record/1950-00412-000
  41. https://www.tandfonline.com/doi/abs/10.1080/01621459.1963.10500849
  42. https://www.laurenceanthony.net/software/antconc/
  43. https://ieeexplore.ieee.org/document/9397187
  44. https://aclanthology.org/2020.acl-main.754.pdf
  45. https://arxiv.org/pdf/2105.02820
  46. https://arxiv.org/pdf/2011.04456
  47. https://ieeexplore.ieee.org/document/1590244
  48. https://arxiv.org/pdf/2011.09682
  49. https://arxiv.org/pdf/1602.00197
  50. https://ieeexplore.ieee.org/document/5696051
  51. https://arxiv.org/pdf/2404.06461
  52. https://bookriot.com/the-gold-bug-edgar-allan-poe/
  53. https://scholarship.claremont.edu/cgi/viewcontent.cgi?article=1751&context=jhm
  54. https://s3-us-west-2.amazonaws.com/screenplays-pdf/Movie-Screenplay-PDFs/Imitation%2BGame.pdf
  55. https://crimereads.com/a-brief-history-of-cryptography-in-crime-fiction/
  56. https://www.theguardian.com/film/2016/nov/28/selma-100-per-cent-historically-accurate-imitation-game-information-is-beautiful
  57. https://yalebooks.yale.edu/2015/01/07/the-real-alan-turing/
  58. https://www.thecollector.com/cryptography-everything-you-need-know/
Add your contribution
Related Hubs
User Avatar
No comments yet.