GloVe

You shall know a word by the company it keeps (Firth, J. R. 1957:11)^[4]

The idea of GloVe is to construct, for each word ${\displaystyle i}$, two vectors ${\displaystyle w_{i},{\tilde {w}}_{i}}$, such that the relative positions of the vectors capture part of the statistical regularities of the word ${\displaystyle i}$. The statistical regularity is defined as the co-occurrence probabilities. Words that resemble each other in meaning should also resemble each other in co-occurrence probabilities.

Let the vocabulary be ${\displaystyle V}$, the set of all possible words (aka "tokens"). Punctuation is either ignored, or treated as vocabulary, and similarly for capitalization and other typographical details.^[1]

If two words occur close to each other, then we say that they occur in the context of each other. For example, if the context length is 3, then we say that in the following sentence

GloVe₁, coined₂ from₃ Global₄ Vectors₅, is₆ a₇ model₈ for₉ distributed₁₀ word₁₁ representation₁₂

the word "model₈" is in the context of "word₁₁" but not the context of "representation₁₂".

A word is not in the context of itself, so "model₈" is not in the context of the word "model₈", although, if a word appears again in the same context, then it does count.

Let ${\displaystyle X_{ij}}$ be the number of times that the word ${\displaystyle j}$ appears in the context of the word ${\displaystyle i}$ over the entire corpus. For example, if the corpus is just "I don't think that that is a problem." we have ${\displaystyle X_{{\text{that}},{\text{that}}}=2}$ since the first "that" appears in the second one's context, and vice versa.

Let ${\displaystyle X_{i}=\sum _{j\in V}X_{ij}}$ be the number of words in the context of all instances of word ${\displaystyle i}$. By counting, we have$${\displaystyle X_{i}=2\times ({\text{context size}})\times \#({\text{occurrences of word }}i)}$$(except for words occurring right at the start and end of the corpus)

Let $${\displaystyle P_{ik}:=P(k|i):={\frac {X_{ik}}{X_{i}}}}$$be the co-occurrence probability. That is, if one samples a random occurrence of the word ${\displaystyle i}$ in the entire document, and a random word within its context, that word is ${\displaystyle k}$ with probability ${\displaystyle P_{ik}}$. Note that ${\displaystyle P_{ik}\neq P_{ki}}$ in general. For example, in a typical modern English corpus, ${\displaystyle P_{{\text{ado}},{\text{much}}}}$ is close to one, but ${\displaystyle P_{{\text{much}},{\text{ado}}}}$ is close to zero. This is because the word "ado" is almost only used in the context of the archaic phrase "much ado about", but the word "much" occurs in all kinds of contexts.

For example, in a 6 billion token corpus, we have

Table 1 of ^[1]

Probability and Ratio	${\displaystyle k={\text{ solid }}}$	${\displaystyle k={\text{ gas }}}$	${\displaystyle k={\text{ water }}}$	${\displaystyle k={\text{ fashion }}}$
${\displaystyle P(k\mid {\text{ ice }})}$	${\displaystyle 1.9\times 10^{-4}}$	${\displaystyle 6.6\times 10^{-5}}$	${\displaystyle 3.0\times 10^{-3}}$	${\displaystyle 1.7\times 10^{-5}}$
${\displaystyle P(k\mid {\text{ steam }})}$	${\displaystyle 2.2\times 10^{-5}}$	${\displaystyle 7.8\times 10^{-4}}$	${\displaystyle 2.2\times 10^{-3}}$	${\displaystyle 1.8\times 10^{-5}}$
${\displaystyle P(k\mid {\text{ ice }})/P(k\mid {\text{ steam }})}$	${\displaystyle 8.9}$	${\displaystyle 8.5\times 10^{-2}}$	${\displaystyle 1.36}$	${\displaystyle 0.96}$

Inspecting the table, we see that the words "ice" and "steam" are indistinguishable along the "water" (often co-occurring with both) and "fashion" (rarely co-occurring with either), but distinguishable along the "solid" (co-occurring more with ice) and "gas" (co-occurring more with "steam").

The idea is to learn two vectors ${\displaystyle w_{i},{\tilde {w}}_{i}}$ for each word ${\displaystyle i}$, such that we have a multinomial logistic regression:$${\displaystyle w_{i}^{T}{\tilde {w}}_{j}+b_{i}+{\tilde {b}}_{j}\approx \ln P_{ij}}$$and the terms ${\displaystyle b_{i},{\tilde {b}}_{j}}$ are unimportant parameters.

This means that if the words ${\displaystyle i,j}$ have similar co-occurrence probabilities ${\displaystyle (P_{ik})_{k\in V}\approx (P_{jk})_{k\in V}}$, then their vectors should also be similar: ${\displaystyle w_{i}\approx w_{j}}$.

Naively, logistic regression can be run by minimizing the squared loss:$${\displaystyle L=\sum _{i,j\in V}(w_{i}^{T}{\tilde {w}}_{j}+b_{i}+{\tilde {b}}_{j}-\ln P_{ij})^{2}}$$However, this would be noisy for rare co-occurrences. To fix the issue, the squared loss is weighted so that the loss is slowly ramped-up as the absolute number of co-occurrences ${\displaystyle X_{ij}}$ increases:$${\displaystyle L=\sum _{i,j\in V}f(X_{ij})(w_{i}^{T}{\tilde {w}}_{j}+b_{i}+{\tilde {b}}_{j}-\ln P_{ij})^{2}}$$where$${\displaystyle f(x)=\left\{{\begin{array}{cc}\left(x/x_{\max }\right)^{\alpha }&{\text{ if }}x<x_{\max }\\1&{\text{ otherwise }}\end{array}}\right.}$$and ${\displaystyle x_{\max },\alpha }$ are hyperparameters. In the original paper, the authors found that ${\displaystyle x_{\max }=100,\alpha =3/4}$ seem to work well in practice.

Once a model is trained, we have 4 trained parameters for each word: ${\displaystyle w_{i},{\tilde {w}}_{i},b_{i},{\tilde {b}}_{i}}$. The parameters ${\displaystyle b_{i},{\tilde {b}}_{i}}$ are irrelevant, and only ${\displaystyle w_{i},{\tilde {w}}_{i}}$ are relevant.

The authors recommended using ${\displaystyle w_{i}+{\tilde {w}}_{i}}$ as the final representation vector for word ${\displaystyle i}$, because empirically it worked better than ${\displaystyle w_{i}}$ or ${\displaystyle {\tilde {w}}_{i}}$ alone.

GloVe can be used to find relations between words like synonyms, company-product relations, zip codes and cities, etc. However, the unsupervised learning algorithm is not effective in identifying homographs, i.e., words with the same spelling and different meanings. This is as the unsupervised learning algorithm calculates a single set of vectors for words with the same morphological structure.^[5] The algorithm is also used by the SpaCy library to build semantic word embedding features, while computing the top list words that match with distance measures such as cosine similarity and Euclidean distance approach.^[6] GloVe was also used as the word representation framework for the online and offline systems designed to detect psychological distress in patient interviews.^[7]

• ELMo
• BERT
• Word2vec
• fastText
• Natural language processing