Sinusoidal Positional Encodings

Take AI Test

by Marcus Transformers

🔍

### Prerequisites
- [Fourier Series as Basis Functions](https://finetuneinterview.com/lessons/fourier-series-as-basis-functions)

## What Problem are we Trying to Solve?

When you first come across Positional Encodings in the paper [Attention is All You Need](https://arxiv.org/pdf/1706.03762) you may ask yourself:

**Why encode position at all?** A Transformer processes its tokens with self attention, and self attention is *permutation equivariant*. The mechanism has no built in notion of which token came first. So "the cat sat" and "sat cat the" look identical to the attention computation without positional encodings. To induce order, we have to add a value denoted as a **positional encoding** that can be learned by the network.

**Why use a sinusoidal form for it?** Note that there are several different methods to perform positional encoding, this method is the one originally used in the *Attention is All You Need* paper. One way to justify this choice is through **Spectral Bias**, also called the **Frequency Principle of Neural Networks**. This states that MLPs (Multilayer Perceptrons, the fully connected layers) naturally learn low frequencies in data first, then high frequencies with more training iterations.

To give an example of this, consider if we had a function
$$f(x) = \sin(x) + 0.3\sin(20x)$$
that we were trying to fit. We can decompose this into
- Low frequency component: $\sin(x)$
- High frequency component: $0.3\sin(20x)$

Often, it's observed that when training an MLP from scratch it'll learn the $\sin(x)$ component first then the $\sin(20x)$ component much later. Although both components are always present, we typically fit the low frequency one first. This typically gives us the largest decrease in error. Additionally, this is a great feature for generalization of neural networks, however it can be problematic in several domains, such as computer vision, where we want *crisp, sharp* images and not *blurry* ones.

The original Transformer paper does not motivate its sinusoidal encoding through spectral bias. They motivated it with that relative positions become linear functions of absolute positions and that the encoding extrapolates to longer sequences than seen in training. Spectral bias is the lens emphasized later in the coordinate MLP literature, such as *NeRF* and *Random Fourier Features*, where feeding raw coordinates into a network produces blurry reconstructions. Spectral bias is used here because it gives clean intuition for *why* frequency based representation is the right tool.

Both the Spectral Bias and relative position motivations give insight to the benefits of this technique. The remedy they point to is the same: instead of asking the network to synthesize oscillations from a raw coordinate, hand it sines and cosines of the position directly as input features. Making that precise, which sines and cosines, at which frequencies, and why the result has the properties attention needs, is the subject of this post.

### Outline
This post builds the Fourier Feature Mapping in two main steps, after a short recap of the Fourier Series. Here we cover:

1. A short recap of the Fourier Series as an orthonormal basis of $L^2[-\pi,\pi]$
2. Representing positions using the Fourier Series, leading to the Fourier Feature Mapping
3. Properties of the Fourier Feature Mapping, focused on Positional Encoding

## Recap: Fourier Series as a Basis

Any well behaved function $f(x)$ on $[-\pi,\pi]$ can be written as a sum of sines and cosines of integer frequencies,

$$ f(x) = \frac{a_0}{2} + \sum_{n=1}^\infty \left[ a_n \cos(nx) + b_n \sin(nx) \right]. $$

The precise statement is that the functions $\{1, \cos(nx), \sin(nx)\}_{n \ge 1}$ form an **orthonormal basis** of $L^2[-\pi,\pi]$, the Hilbert space of square integrable functions on $[-\pi,\pi]$, once each is normalized. They are orthogonal, hence linearly independent, and their closed span is the whole space, so nothing is left out. The coefficients $a_n, b_n$ are then just the projections of $f$ onto those basis functions, exactly as coordinates in $\mathbb{R}^n$ are projections onto the standard basis.

The full treatment lives in the prerequisite [Fourier Series as Basis Functions](https://finetuneinterview.com/lessons/fourier-series-as-basis-functions): what $L^2[-\pi,\pi]$ is, the orthonormal normalization (the trig functions are orthogonal but not unit norm, which is where the coefficient's $\tfrac{1}{\pi}$ factor comes from), and the orthogonality plus completeness proof. For positional encoding we will not need the coefficient values themselves, since we set them all to $1$, only the fact that these sines and cosines form a basis.

To see the power of the Fourier Series to approximate functions, see the PyApplet below.

## Fourier Series Representing Position

The Fourier Series represents a *function* by its list of coefficients, just as a point in $\mathbb{R}^3$ is the list of coordinates multiplying the standard basis vectors:

$$ v = 1 \cdot [1,0,0]^T + 2 \cdot [0,1,0]^T + 3 \cdot [0,0,1]^T $$

is the point $[1,2,3]^T$, and in the same way the spectral bias example from the introduction, $f(x) = \sin(x) + 0.3\sin(20x)$, is the coefficient list with $b_1 = 1$, $b_{20} = 0.3$, and every other coefficient zero. We are simply re-expressing the function in terms of its Fourier Basis, just like we restate a coordinate in $\mathbb{R}^3$ in terms of its Standard Basis.

A position, however, is a number, not a function, so there are no coefficients to read off. Instead we run the construction in the other direction: evaluate each basis function $\cos(\omega_i x)$ and $\sin(\omega_i x)$ at a particular $x$ (the position), with all coefficients set to $1$, and **stack the evaluated values into a vector** rather than summing them. The coefficients are set to $1$ since we are interested in evaluating the position in this already descriptive basis, and we keep the components side by side exactly because we want the network to see the individual frequencies. It can then read low and high frequencies directly off the vector instead of synthesizing them, which is the remedy to spectral bias promised in the introduction.

The difference between reconstruction and feature mapping is:
* Reconstruction sums the basis functions back into one scalar
* Feature mapping concatenates chosen basis functions, evaluated at a point, into a vector

### A Position in $[-\pi, \pi)$

Let the position be a real number $x \in [-\pi, \pi)$. We pick a set of integer frequencies and evaluate every sine and cosine at $x$, fixing all coefficients to $1$:

$$ \phi(x) = \big[\, \cos(0\cdot x),\; \cos(x), \sin(x),\; \cos(2x), \sin(2x),\; \dots,\; \cos(Mx), \sin(Mx) \,\big]. $$

The first entry is the constant $1$ from $\cos(0x)$, and there is no $\sin(0x)$ entry since it is identically zero. Truncating at frequency $M$ gives a finite vector of length $2M + 1$. Each coordinate reports "how much of this frequency is present at this location", and together they pin down where $x$ sits.

If your positions live in some other interval $[a, b)$ rather than $[-\pi, \pi)$, you do not need new machinery. Rescale with the affine map
$$ x \mapsto \frac{2\pi (x - a)}{b - a} - \pi, $$
which carries $[a,b)$ onto $[-\pi, \pi)$. This only rescales the effective frequencies, so the construction is unchanged. See the [Appendix](#appendix) for the derivation of this map and what it does to the frequencies.

### A Discrete Set $\{0, 1, \dots, N-1\}$ with the Discrete Fourier Series

Token positions in a Transformer are not continuous; they are the integers $0, 1, \dots, N-1$. The natural object here is the **Discrete Fourier Series**, which is what you get by restricting the Fourier construction to functions sampled at these $N$ points.

On $N$ samples there are $N$ distinct frequencies,
$$ \omega_k = \frac{2\pi k}{N} \quad \text{for } k = 0, 1, \dots, N-1. $$

For a real valued function the frequency $k$ and the frequency $N - k$ carry the same information. On integer samples they give the same cosine and equal but opposite sines,
$$ \cos\!\big(\tfrac{2\pi (N-k)}{N} n\big) = \cos\!\big(\tfrac{2\pi k}{N} n\big), \qquad \sin\!\big(\tfrac{2\pi (N-k)}{N} n\big) = -\sin\!\big(\tfrac{2\pi k}{N} n\big), $$
so the upper half of the range, $k = N/2 + 1, \dots, N-1$, are mirror images of the lower half and do not add new information. This is why we only need $k = 0, 1, \dots, N/2$.

Within that reduced range the two endpoints are special, because their sine vanishes: at $k = 0$ we have $\sin(0) = 0$, and at $k = N/2$ we have $\sin(\pi n) = 0$ for every integer $n$. So $k = 0$ and $k = N/2$ each contribute only a cosine, while every interior frequency $k = 1, \dots, N/2 - 1$ contributes a cosine and a sine. Writing this out, the encoding for an even $N$ is

$$ PE_{\mathrm{DFS}}(pos) = \Big[\, \cos(0),\; \sin\!\big(\tfrac{2\pi}{N}pos\big), \cos\!\big(\tfrac{2\pi}{N}pos\big),\; \dots,\; \sin\!\big(\tfrac{2\pi(\tfrac{N}{2}-1)}{N}pos\big), \cos\!\big(\tfrac{2\pi(\tfrac{N}{2}-1)}{N}pos\big),\; \cos(\pi\, pos) \,\Big]. $$

Here $PE$ is Positional Encoding and $DFS$ is Discrete Fourier Series. The final term is the **Nyquist** term $\cos(\pi\, pos)$, defined as the highest frequency the sampling can represent, obtained from $\omega_{N/2} = \tfrac{2\pi}{N}\cdot \tfrac{N}{2} = \pi$.

To determine the dimensionality of the vector we count the number of elements: 
- One constant
- Two each ($\sin$ and $\cos$) for the $\tfrac{N}{2}-1$ interior frequencies
- One Nyquist cosine
    - Note: The sine for $sin(\pi\ pos) = 0$ since $sin(\pi n) = 0$, $\forall n \in \mathbb{Z}$

Giving us
$$ 1 + 2\left(\tfrac{N}{2} - 1\right) + 1 = N $$
basis functions for $N$ positions. That is exactly a complete basis for functions on $\{0, \dots, N-1\}$, the discrete mirror of the continuous result from the last section.

### The Fourier Feature Mapping as a Subsampling of the Discrete Fourier Series

The Discrete Fourier Series is the complete basis for $N$ points. However, the encoding dimension $d$ is usually much smaller than the sequence length $N$, so we select a subset of the frequencies. Choosing $d/2$ of them gives a **Fourier Feature Mapping**, an encoding of length $d$:

$$ \phi(pos) = \big[\, \sin(\omega_0\, pos), \cos(\omega_0\, pos),\; \dots,\; \sin(\omega_{d/2 - 1}\, pos), \cos(\omega_{d/2 - 1}\, pos) \,\big] \in \mathbb{R}^d. $$

We can choose how to space the chosen frequencies $\omega_i$. The Transformer picks a geometric progression,
$$ \omega_i = \frac{1}{10000^{2i/d}}, \qquad i = 0, 1, \dots, \tfrac{d}{2} - 1, $$
which written out gives the encoding from *Attention is All You Need*:
$$ PE(pos, 2i) = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE(pos, 2i+1) = \cos\!\left(\frac{pos}{10000^{2i/d}}\right). $$

The wavelengths run from $2\pi$ at the highest frequency up to about $2\pi \cdot 10,000$ at the lowest, geometrically spaced in between.

Collecting the chosen frequencies into a matrix $B$ recovers the compact form often seen in the literature
$$ \phi(pos) = [\sin(B\, pos), \cos(B\, pos)]. $$
For a scalar position, $B$ is just the column vector that stacks the frequencies, so $B\, pos$ scales the position by each one. With a small encoding dimension $d = 4$ we have two Transformer frequencies, $\omega_0 = 10000^{0} = 1$ and $\omega_1 = 10000^{-1/2} = 0.01$, giving

$$ B = \begin{pmatrix} 1 \\ 0.01 \end{pmatrix}, \qquad B\, pos = \begin{pmatrix} pos \\ 0.01\, pos \end{pmatrix}$$

$$\phi(pos) = \big[\, \sin(pos),\; \sin(0.01\, pos),\; \cos(pos),\; \cos(0.01\, pos) \,\big]. $$

Applying sine and cosine entrywise to the vector $B\, pos$ and stacking the results gives the length $d$ encoding. Note that $[\sin(B\, pos), \cos(B\, pos)]$ places all the sines first and all the cosines second, which is the same set of numbers as the interleaved $[\sin(\omega_0\, pos), \cos(\omega_0\, pos), \dots]$ ordering above, just permuted.

### Another choice of frequencies: NeRF

Depending on the application, we may want to space $\omega_i$ differently. NeRF makes a different choice for the same feature mapping, using frequencies that are powers of two,
$$ \omega_i = 2^i \pi, \qquad i = 0, 1, \dots, L-1, $$
so the encoding of a coordinate $p$ is
$$ \gamma(p) = \big[\, \sin(2^0 \pi p), \cos(2^0 \pi p),\; \dots,\; \sin(2^{L-1} \pi p), \cos(2^{L-1} \pi p) \,\big]. $$

Both choices are geometric progressions, equally spaced on a log axis, which is what "log spaced" means below; they differ only in the ratio and the direction. The Transformer frequencies $10000^{-2i/d}$ shrink from $1$ down toward $1/10000$ as $i$ grows, while NeRF's $2^i \pi$ grow by a factor of two at each step. NeRF also applies this mapping to each input coordinate separately, with inputs normalized to $[-1, 1]$, and typically uses $L = 10$ for spatial coordinates. When the coordinate is itself multidimensional, as in NeRF where the input lives in $\mathbb{R}^3$, the matrix $B$ from the compact form becomes a genuine $m \times 3$ matrix rather than a column vector, and $B x$ mixes the input coordinates before taking sines and cosines.

The same feature mapping is common across domains, and the main difference between methods is how the frequencies in $B$ are chosen and spaced.

| Method | Frequencies |
|--------|-------------------------|
| Transformer positional encoding | Fixed geometric (log spaced) frequencies, base $10000$ |
| NeRF | Fixed geometric (log spaced) frequencies, powers of two |
| Random Fourier Features | Random Gaussian frequencies |
| Diffusion timestep embeddings | Fixed geometric (log spaced) frequencies |
| RoPE | Fixed frequencies applied as rotations in query and key space |
| Learned Fourier features | Learned during training |

## Properties of the Positional Encoding

We can now derive properties of the encoding, working directly from
$$ \phi(pos) = \big[\, \sin(\omega_0\, pos), \cos(\omega_0\, pos),\; \dots,\; \sin(\omega_{d/2-1}\, pos), \cos(\omega_{d/2-1}\, pos) \,\big] $$
$$\omega_i = 10000^{-2i/d}. $$

### Every position has a unique representation, over practical lengths

Look at the lowest frequency component, $i = \tfrac{d}{2} - 1$, where
$$ \omega_{\min} = 10000^{-(d-2)/d} \approx \frac{1}{10000}. $$
The pair $\big(\sin(\omega_{\min} pos), \cos(\omega_{\min} pos)\big)$ encodes the angle $\omega_{\min}\, pos$ modulo $2\pi$, and this angle is distinct for every position until it completes a full turn. That happens once
$$ pos = \frac{2\pi}{\omega_{\min}} \approx 2\pi \cdot 10000 \approx 62{,}832. $$
So for any sequence shorter than roughly sixty thousand tokens, the lowest frequency pair alone already assigns a distinct angle to each position, and the full encoding is therefore unique. This is why the property holds "over practical sequence lengths" rather than for all integers.

### Relative positions are linear transformations of absolute positions

Fix an offset $k$. For a single frequency $\omega$, the angle addition identities give
$$ \begin{pmatrix} \sin(\omega(pos + k)) \\ \cos(\omega(pos + k)) \end{pmatrix} = 
\begin{pmatrix} 
\cos(\omega k) & \sin(\omega k) \\ 
-\sin(\omega k) & \cos(\omega k) 
\end{pmatrix} 
\begin{pmatrix} 
\sin(\omega\, pos) \\ \cos(\omega\, pos) 
\end{pmatrix}. $$

As we can see from looking at the calculation for $\sin(\omega(pos + k))$
$$\sin(\omega(pos + k)) = \cos(\omega k)\sin(\omega\, pos) + \sin(\omega k)\cos(\omega\, pos) $$

which is the identity
$$\sin(a+b) = \sin a \cos b + \cos a \sin b$$

Similarly, $\cos(\omega(pos + k))$ utilizes the cosine identity
$$\cos(\omega(pos + k)) = -\sin(\omega k) sin(\omega\ pos) + \cos(\omega k) cos(\omega\ pos) $$

with the cosine identity
$$\cos(a+b) = \cos a \cos b - \sin a \sin b $$

Stacking these $2 \times 2$ blocks for every frequency gives a single block diagonal matrix $M_k$ with
$$ \phi(pos + k) = M_k\, \phi(pos) $$

The key is that $M_k$ depends only on the offset $k$, not on $pos$. So shifting by a fixed relative distance is the *same* linear map regardless of where you start, which is precisely what lets attention compare positions by their relative offset.

### Inner products depend only on relative distance

Compute the inner product of two encodings separated by offset $k$:

$$ \langle \phi(pos), \phi(pos + k) \rangle = \sum_{i=0}^{d/2 - 1} \Big[ \sin(\omega_i\, pos)\sin(\omega_i(pos+k)) + \cos(\omega_i\, pos)\cos(\omega_i(pos+k)) \Big] $$

Each bracket is the cosine difference identity 
$$\cos(a - b) = \cos a \cos b + \sin a \sin b$$

with 
- $a = \omega_i(pos+k)$ 
- $b = \omega_i\, pos$

so it simplifies to $\cos(\omega_i k)$. Therefore
$$ \langle \phi(pos), \phi(pos + k) \rangle = \sum_{i=0}^{d/2 - 1} \cos(\omega_i k) $$

This depends on $k$ and not on $pos$. Because cosine is even, it is also symmetric when $k \mapsto -k$. Thus the similarity between two positions is a function of their distance.

### Smoothness: Similar Positions have Similar Embeddings

First note the encoding has constant norm, since each frequency pair contributes $\sin^2 + \cos^2 = 1$:

$$ \|\phi(pos)\|^2 = \sum_{i=0}^{d/2-1} sin(\omega_i\ pos)^2 + cos(\omega_i\ pos)^2 $$
$$ \|\phi(pos)\|^2 = \sum_{i=0}^{d/2-1} 1 = \frac{d}{2}$$

Every position as a point on the L2 Sphere. With the inner product result, the squared distance between two encodings is

$$ \| \phi(pos) - \phi(pos + k)\|^2 = \|\phi(pos)\|^2 + \|\phi(pos+k)\|^2 - 2\langle \phi(pos), \phi(pos+k)\rangle $$
$$ = \frac{d}{2} + \frac{d}{2} - 2\sum_{i=0}^{d/2-1}\cos(\omega_i k)$$
$$ = d - 2\sum_{i=0}^{d/2-1}\cos(\omega_i k) $$
$$= 2\sum_{i=0}^{d/2-1}\big(1 - \cos(\omega_i k)\big) $$

For small offsets, expanding 
$$1 - \cos(\omega_i k) \approx \tfrac{1}{2}\omega_i^2 k^2$$

shows the distance grows with shape $k^2$. Thus nearby positions are close, and the encoding is a smooth (infinitely differentiable) function of position, built with sines and cosines.

## Adding Positional Encodings to Tokens

The way this is utilized in the *Attention is All You Need Paper* is to simply elementwise add this Sinusoidal Positional Encoding to the Token, both of dimension $d$.

For example, for tokens $a,b \in \mathbb{R}^d$ with their positional encodings also in $PE_a, PE_b \in \mathbb{R}^d$ we produce position aware token vectors $\tilde{a}, \tilde{b} \in \mathbb{R}^d$

$$ \tilde{a} = a + PE_a, \qquad \tilde{b} = b + PE_b. $$

Componentwise, with $j = 0, 1, \dots, d-1$ indexing the embedding dimension, we have

$$ \tilde{a} = \begin{pmatrix} a_0 \\ a_1 \\ \vdots \\ a_{d-1} \end{pmatrix} + \begin{pmatrix} PE_{a,0} \\ PE_{a,1} \\ \vdots \\ PE_{a,d-1} \end{pmatrix} = \begin{pmatrix} a_0 + PE_{a,0} \\ a_1 + PE_{a,1} \\ \vdots \\ a_{d-1} + PE_{a,d-1} \end{pmatrix}. $$

Because addition preserves the dimension, $\tilde{a}$ still lives in $\mathbb{R}^d$ and can be passed directly into attention.

To see what information is preserved when we inner product $\tilde{a}$ by $\tilde{b}$ similar to how keys are inner producted (dot producted) by queries in the attention mechanism, let's walk through an example, where we consider only one element from $\tilde{a}$ and $\tilde{b}$ at the same index

$$ \tilde{a}_i = a_i + PE_{a,i}$$
$$ \tilde{b}_i = b_i + PE_{b,i}$$

when multiplied we get
$$ \tilde{a}_i \cdot \tilde{b}_i  = (a_i + PE_{a,i})(b_i + PE_{b,i}) $$
Each of the four terms are
$$ \tilde{a}_i \cdot \tilde{b}_i  = a_i b_i + a_i PE_{b,i} + b_i PE_{a,i} + PE_{a,i}PE_{b,i}$$

As we can see we have four terms, which can be described as:
1. $a_i b_i$ how the tokens relate to each other 
2. $a_i PE_{b,i}$ how the token $a$ relates to the position of $b$
3. $b_i PE_{a,i}$ how the token $b$ relates to the position of $a$
4. $PE_{a,i}PE_{b,i}$ how the position of $a$ relates to the position of $b$

For term 4, i.e. how the positions relate to each other, we have two cases. Since due to the dot product, we only dot the same indices of the position vector together in case 4, $PE_{a,i}PE_{b,i}$, we have that we're either 
- Case 1: Multiplying Two Sines 
- Case 2: Multiplying Two Cosines

#### Case 1: Multiplying Two Sines

For Case 1, we have 
$$ PE_{a,i}PE_{b,i} = sin(\omega_i\ pos_a) sin(\omega_i\ pos_b)$$

Using the trig identity 
$$ sin(a) sin(b) = \frac{1}{2} (cos(a-b) - cos(a+b))$$

we get that both positions can be related to each other through a difference and addition fed into the cosine
$$ PE_{a,i}PE_{b,i} = \frac{1}{2} (cos(\omega_i (pos_a-pos_b)) - (cos (\omega_i (pos_a+pos_b)))$$

#### Case 2: Multiplying Two Cosines 
For Case 2, we have 
$$ PE_{a,i}PE_{b,i} = cos(\omega_i\ pos_a) cos(\omega_i\ pos_b)$$

Using the trig identity 
$$ \cos(a)\cos(b)=\frac{1}{2}\left(\cos(a-b)+\cos(a+b)\right) $$

we get that both positions can be related to each other through a difference and addition fed into the cosine

$$ PE_{a,i}PE_{b,i} = \frac{1}{2} (cos(\omega_i (pos_a-pos_b)) + (cos (\omega_i (pos_a+pos_b)))$$

This is one of the main reasons why Sinusoidal Positional Encodings are used for Transformers, since given the dot product of the keys and queries, we can encode information of how the tokens and their positions relate to each other. Furthermore it is easy to add the positional encoding vector to the token vector, which is great for implementation and debugging.

---

## Appendix

### Deriving the affine map from $[a, b)$ to $[-\pi, \pi)$

This is just min max normalization. To send a value onto $[0, 1)$ we shift by the minimum coordinate $a$ and scale by the width $b - a$:
$$ \frac{x - a}{b - a} \in [0, 1). $$
<center>
Min max scaling between $[0,1]$ for input $x$
</center>

To land on a general target interval $[c, d)$ instead of $[0, 1)$, we scale by the target width $d - c$ and shift by the target minimum $c$:
$$ x' = c + \frac{(x - a)(d - c)}{b - a}. $$
This is the general min max form: any source interval $[a, b)$ rescaled onto any target $[c, d)$.

For our case the target is $[-\pi, \pi)$, so $c = -\pi$ and $d = \pi$, giving a width of $d - c = 2\pi$:
$$ x' = -\pi + \frac{2\pi\,(x - a)}{b - a} = \frac{2\pi\,(x - a)}{b - a} - \pi, $$
which is the map used in the main text.