Three generations of embeddings, and what each one measured

Barakaeli Lawuo, May 13, 2026

In August 2019 Nils Reimers and Iryna Gurevych put a price on a question every search system asks: which two sentences in this pile mean the same thing? With BERT, the strongest sentence-pair model of the time, both sentences go into the network together and it predicts a score. For a collection of 10,000 sentences that means n(n−1)/2 = 49,995,000 passes through the network, which they measured at about 65 hours on a V100 GPU.¹ Their modified network, Sentence-BERT (SBERT), ran each sentence through once to get a vector. Computing all 10,000 vectors took about 5 seconds and comparing them with cosine similarity took about 0.01 seconds.¹ The same paper notes that matching one new question against Quora's more than 40 million existing ones would need over 50 hours per query with pairwise BERT.¹

That gap is the whole commercial case for embeddings. Comparing two lists of numbers is cheap, and running a large network on every pair is not. The catch is that the cheap comparison is only worth anything if the geometry of the vectors actually tracks meaning, and what counted as "tracks meaning" changed a lot between 2013 and 2022. Three papers mark the generations: word2vec made word vectors cheap to train, SBERT made sentence vectors from a pretrained transformer usable, and MTEB showed that no single embedder was best at everything.

Embedding

A fixed-length list of numbers (a vector) that a model assigns to a word, sentence, or passage. Texts with similar meaning are supposed to get vectors that point in similar directions.

Cosine similarity

The cosine of the angle between two vectors. It is 1 when they point the same way, 0 when they are at right angles, and ignores their lengths.

Cross-encoder

A model that reads two texts together and outputs one score for the pair. Accurate, but it produces no reusable vector, so every new pair costs a full forward pass.

Bi-encoder

A model that encodes each text on its own into a vector. Vectors are computed once and compared with a cheap similarity function. SBERT is one.

Spearman correlation

A score from −1 to 1 for how well two rankings agree. Embedding papers use it to compare the ranking of pairs by cosine similarity against the ranking by human similarity labels.

2013: word vectors trained by predicting neighbors

Before word2vec, many NLP systems treated words as atomic units: indices into a vocabulary, with no notion that one word is similar to another.² Neural language models already learned word vectors, but most of their compute went into a dense nonlinear hidden layer. Tomas Mikolov and colleagues at Google removed that layer and proposed two log-linear models. CBOW predicts a word from its surrounding words. Skip-gram does the reverse: it uses the current word to predict words within a window before and after it.² A follow-up paper wrote the skip-gram training goal down explicitly:³

Read it term by term. The training text is a sequence of $T$ words $w_1, \dots, w_T$. For each position $t$, the inner sum runs over neighbors up to $c$ words away, where $c$ is the size of the training context. $p(w_{t+j}\mid w_t)$ is the probability the model gives to seeing that neighbor, given the center word. Each word has two vectors: an input vector $v_w$, used when the word is the center, and an output vector $v'_w$, used when it is the thing being predicted. The dot product ${v'_{w_O}}^{\top} v_{w_I}$ is large when the two vectors point the same way, and the softmax turns those dot products into probabilities over the vocabulary of $W$ words.³ So a word's vector is pushed toward the output vectors of the words it tends to appear near, and words that share neighbors end up pointing in similar directions.

The denominator is the problem. It sums over the whole vocabulary, which the authors put at $10^5$ to $10^7$ terms, for every single training example.³ Their fix, negative sampling, replaces each $\log p(w_O\mid w_I)$ with a small logistic-regression task: raise $\log\sigma({v'_{w_O}}^{\top} v_{w_I})$ for the real neighbor and lower the score of $k$ random noise words drawn from the unigram distribution raised to the 3/4 power. Here $\sigma$ is the sigmoid function. They found $k$ of 5 to 20 useful for small datasets and 2 to 5 enough for large ones.³

The measurement they built for this was the analogy test. To answer "what is to small as biggest is to big," compute vector(biggest) − vector(big) + vector(small), then return the word whose vector is closest by cosine distance, excluding the three question words.² Their test set had 8,869 semantic questions (such as capital cities and currencies) and 10,675 syntactic ones (such as plurals and past tenses). Only an exact match counted, so a synonym scored as a miss, and the authors said 100% was likely impossible.²

On the same 320 million words of training data with 640-dimensional vectors, vectors from a recurrent language model got 9% of the semantic questions right; skip-gram got 55%.² Scaled up to Google News (about 6 billion tokens) with 1,000 dimensions, skip-gram reached 66.1% semantic, 65.1% syntactic and 65.6% overall, trained in 2.5 days on about 125 CPU cores. A feedforward neural language model with 100-dimensional vectors reached 50.8% and took 14 days on about 180 cores.² In the follow-up, negative sampling with 15 noise words scored 61% overall against 47% for a Huffman-tree hierarchical softmax, and discarding frequent words (subsampling) cut its training time from 97 to 36 minutes with the same total accuracy.³

The same paper also names what this generation cannot do. It calls this an inherent limitation of word representations: "their indifference to word order and their inability to represent idiomatic phrases." The meanings of "Canada" and "Air" do not combine into "Air Canada."³ Their workaround was to detect frequent phrases and give each its own token.³ Each word or phrase still got exactly one vector, whatever sentence it appeared in.

2019: BERT could compare sentences but not embed them

Transformers such as BERT use self-attention to make each word's representation depend on its context, which word vectors like GloVe lack.⁴ That addresses the one-vector-per-word problem, but BERT was built as a cross-encoder: it gives one score per pair and no standalone sentence vector.¹ People worked around this by feeding in a single sentence and either averaging the output layer or taking the output of the special [CLS] token. Reimers and Gurevych measured both. Across seven semantic textual similarity (STS) tasks, averaged BERT outputs scored 54.81 and the [CLS] vector 29.19 (Spearman correlation × 100). Plain averaged GloVe word vectors, a word-vector method from 2014, scored 61.32.¹ Out of the box, BERT's vector space was worse for cosine similarity than static word vectors.

SBERT keeps BERT and changes the training. A pooling step (by default, the mean of all output vectors) turns BERT's per-token outputs into one fixed-size vector. Two copies of the network with shared weights, a siamese network, encode sentence A and sentence B separately into vectors $u$ and $v$. When training on labeled sentence pairs, the two vectors feed a small classifier:¹

Here $u$ and $v$ are the two sentence embeddings, each with $n$ numbers. $|u-v|$ is their element-wise absolute difference. The three are concatenated into one vector of length $3n$, which a trainable weight matrix $W_t$ maps to $k$ scores, one per label, and the softmax turns those into probabilities $o$.¹ For training data they used 570,000 SNLI and 430,000 MultiNLI sentence pairs labeled contradiction, entailment or neutral, trained for one epoch.¹ In the ablation, the difference term $|u-v|$ was the most important component, and adding the element-wise product $u * v$ made results worse.¹ The classifier is thrown away after training. At inference only $u$, $v$ and cosine similarity are used, so the loss teaches the encoder to put sentences with the same meaning close together and sentences with different meaning further apart.¹ A second, regression objective trains on the cosine similarity of $u$ and $v$ directly, with mean squared error against a gold score.¹

Trained only on NLI data, SBERT-NLI-large averaged 76.55 on the seven STS tasks. The paper reports that as 11.7 points above InferSent and 5.5 points above Universal Sentence Encoder, two earlier sentence embedders it names as state of the art.¹ Averaged BERT outputs had managed 54.81, so the same underlying network gained more than 20 points from a change of objective alone.

The paper is also clear about what the speedup costs. When both kinds of model were trained on NLI and then on the STS benchmark, the BERT-large cross-encoder scored 88.77 on the benchmark's test set and SBERT-large scored 86.10.¹ On argument pairs from an unseen debate topic, the gap was about 7 points of Spearman correlation: 60.34 for the BERT-large cross-encoder against 53.10 for SBERT-large.¹ Their explanation is that BERT can use attention to compare the two sentences word by word, while SBERT has to place each sentence in the vector space alone, without seeing the other one.¹ A bi-encoder buys its speed by committing to a vector before it knows what it will be compared with.

2022: 58 datasets and no model on top of all of them

By 2022 there were dozens of sentence embedders, and most were judged on a narrow slice of tasks. The MTEB authors point out that SBERT itself was evaluated only on STS and classification, and that STS is known to correlate poorly with other real-world uses.⁴ The Massive Text Embedding Benchmark (MTEB) spans 8 task types, 58 datasets and 112 languages, and the paper benchmarked 33 models on it.⁴ Almost every task reduces to one operation between two vectors:

$u$ and $v$ are two embeddings with $n$ components each, and $u_i$ is the $i$-th number in $u$. The top, $u\cdot v$, is the dot product: multiply matching components and add them up. The bottom divides by the two lengths $\lVert u\rVert$ and $\lVert v\rVert$, so only direction matters. In retrieval, each query and each document is embedded, documents are ranked by cosine similarity to the query, and nDCG@10, a score for how well the top ten results are ordered, is the main metric. In STS, the main metric is the Spearman correlation between cosine similarities and human labels. Classification is the exception: a logistic-regression classifier is trained on the embeddings and scored on accuracy.⁴

The headline finding is in the paper's abstract: no particular text embedding method dominates across all tasks.⁴ ST5-XXL, built on the encoder of T5 and fine-tuned on NLI, had the best overall average at 59.51 and the best classification and STS scores. It lost retrieval to SGPT-5.8B-msmarco, a GPT-style decoder fine-tuned for retrieval on MS MARCO, by 42.24 to 50.25.⁴ Models tuned for STS did badly at retrieval, and retrieval models did badly at STS. The authors connect this to asymmetry: retrieval compares two different kinds of text, short queries and longer documents, while STS compares two texts of the same kind.⁴ MPNet, almost 50 times smaller than ST5-XXL, matched it on clustering (43.69 against 43.71) and led reranking at 59.36.⁴

Two other patterns in that table cut against intuition. First, the gap between self-supervised and supervised models was still large. Plain BERT scored 10.59 on retrieval, lower than GloVe's 21.62.⁴ The authors conclude that language models still seemed to need supervised fine-tuning to produce competitive embeddings.⁴ Second, scores rose strongly with model size, but the largest models were also the slowest, and SGPT-5.8B's high-dimensional vectors need more storage.⁴ Their efficiency analysis sorts models into three groups: word-embedding models like GloVe for maximum speed, the fine-tuned MPNet and MiniLM models for a balance of speed and quality, and GTR-XXL, ST5-XXL or SGPT-5.8B when quality matters more than latency.⁴

What the Google News vectors also learned

Geometry that picks up "Paris is to France as Rome is to Italy" from co-occurrence will pick up whatever else co-occurs. In 2016 Tolga Bolukbasi and colleagues studied the publicly released word2vec vectors trained on Google News: 3 million English words and terms in 300 dimensions.⁵ The same analogy arithmetic that answers man:king :: woman:x with queen also returns man:computer programmer :: woman:homemaker, and says a father is to a doctor as a mother is to a nurse.⁵ One might have hoped for little gender bias, the authors write, since many of the articles' authors are professional journalists.⁵

They made the bias measurable. First they took the difference vectors of ten gender pairs such as she and he, and found that one principal component explained most of their variance. They called that unit vector the gender direction $g$.⁵ Occupation words projected onto the she−he axis correlated with crowd workers' stereotype ratings at Spearman ρ = 0.51. The projections were also highly consistent between the word2vec vectors and GloVe vectors trained on a web crawl (ρ = 0.81), which suggests the bias was not an artifact of one corpus or one algorithm.⁵ For a set $N$ of words that ought to be gender neutral, they defined direct bias as the average alignment with that direction:

$\vec{w}$ is a word's vector, $|N|$ is the number of neutral words, and $|\cos(\vec{w}, g)|$ is how strongly that word leans toward either end of the gender direction. With $c = 1$ the measure is a plain average. Over 327 occupation words, the Google News vectors scored 0.08, which the authors read as many occupations carrying a substantial gender component.⁵ In a crowd study, workers judged 19% of the top 150 generated she-he analogies to be stereotypes. After the authors' hard-debiasing step, which removes the gender component from words that should be neutral, the figure fell to 6%.⁵ The puzzle he:doctor :: she:x went from nurse to physician.⁵ Scores on two word-similarity benchmarks and an analogy benchmark barely moved: 62.3 to 62.4, 54.5 to 54.1, and 57.0 to 57.0.⁵

My reading, not a claim from any of these papers: there is no separate channel for bias. The offset arithmetic that earned skip-gram its 65.6% is the same operation that produced homemaker. The bias measurements above are for static word vectors; none of the five papers cited here measures gender bias in SBERT-style sentence encoders or in the MTEB models, so it is unknown from these sources how much of it survives into them.

The average score leans toward the big task types

The MTEB appendix lists the benchmark's own limitations, and one of them bears directly on how people use it. Task types have different numbers of datasets: summarization has one, pair classification three, retrieval fifteen, classification twelve and clustering eleven.⁴ The overall MTEB score averages over datasets, so, in the authors' words, it is biased toward the task types with many datasets, notably retrieval, classification and clustering.⁴ The same appendix notes that the longest texts in MTEB are a few hundred words, that retrieval and clustering are English-only, and that there are no code datasets.⁴ A model that sits at the top of the average may therefore be the best retrieval and classification model on short English text, and nothing more is guaranteed by that one number.