Transformer (machine learning model)

A transformer is a type of deep learning model architecture designed to process sequential or structured data using mechanisms based primarily on attention, particularly self-attention.

Transformers were introduced in the 2017 paper "Attention Is All You Need" by Google researchers. The architecture was initially developed for sequence-to-sequence tasks such as machine translation and subsequently became widely used in natural language processing, computer vision, speech processing, and other machine-learning applications.

A simplified representation is:

$\text{Input Sequence}\to \text{Embedding}\to \text{Self-Attention}\to \text{Feed-Forward Processing}\to \text{Output}$

Transformer models can process relationships between elements of a sequence without relying exclusively on sequential recurrence.

Architecture

A transformer typically consists of multiple layers containing attention and feed-forward components.

A simplified transformer layer can be represented as:

$\text{Input}\to \text{Self-Attention}\to \text{Feed-Forward Network}\to \text{Output}$

Modern transformer architectures may also include residual connections, normalization layers, positional representations, and other components.

The original transformer architecture contained separate encoder and decoder stacks.

Encoder and decoder

The original architecture consists of:

• an encoder, which processes the input sequence;
• a decoder, which generates the output sequence.

A simplified sequence-to-sequence representation is:

$\text{Input Sequence}\to \text{Encoder}\to \text{Decoder}\to \text{Output Sequence}$

The encoder converts the input into contextual representations that can be used by the decoder.

The decoder generates output tokens while attending to previously generated tokens and, in the original architecture, representations produced by the encoder.

Attention mechanism

Attention allows a model to assign different weights to elements of an input sequence when constructing a representation.

A simplified attention operation is:

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

where:

• $Q$ = query matrix;
• $K$ = key matrix;
• $V$ = value matrix;
• $d_k$ = dimensionality of the key vectors.

The scaling factor $\sqrt{d_k}$ helps control the magnitude of the dot products before the softmax operation.

Self-attention

Self-attention is an attention mechanism in which the queries, keys, and values are derived from the same input sequence.

A simplified representation is:

$Q=X{W}^Q,K=X{W}^K,V=X{W}^V$

where:

• $X$ = input sequence representation;
• $Q$ = query matrix;
• $K$ = key matrix;
• $V$ = value matrix;
• $W^Q$, $W^K$, and $W^V$ = learned projection matrices.

The self-attention operation can then be represented as:

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

This allows each element of a sequence to incorporate information from other elements.

For example, in a sentence, a token can assign greater attention to other tokens that provide relevant contextual information.

Multi-head attention

Transformers commonly use multi-head attention, in which several attention operations are performed in parallel.

For each attention head:

${\text{head}}_i=\mathrm{Attention}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$

where $W_i^Q$, $W_i^K$, and $W_i^V$ are learned projection matrices for head $i$.

The outputs of the attention heads are concatenated and projected:

$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W^O$

where:

• $h$ = number of attention heads;
• $W^O$ = output projection matrix.

Multiple attention heads allow the model to learn different relationships or representations from the same input sequence.

Positional information

Because self-attention does not inherently impose a sequential processing order, transformer models require mechanisms for representing positional information.

A simplified representation is:

$\text{Input Representation}=\text{Token Representation}+\text{Positional Representation}$

The original transformer architecture used sinusoidal positional encodings.

For position $pos$ and dimension $i$, the original formulation includes:

$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

and:

$P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

Modern transformer architectures may instead use learned positional embeddings or other positional encoding mechanisms.

Feed-forward network

Transformer layers commonly contain a position-wise feed-forward network following the attention mechanism.

A simplified formulation is:

$\mathrm{FFN}(x)=\sigma (x{W}_1+b_1)W_2+b_2$

where:

• $x$ = input representation;
• $W_1$ and $W_2$ = learned weight matrices;
• $b_1$ and $b_2$ = bias vectors;
• $\sigma$ = activation function.

The original transformer architecture used a ReLU activation function.

Residual connections and normalization

Transformer layers commonly use residual connections and normalization.

A simplified residual relationship is:

$\text{Output}=\text{Input}+\text{Transformation}(\text{Input})$

Residual connections help information and gradients propagate through deep networks.

Normalization is used to stabilize and improve the training of deep transformer networks.

Tokenization and embeddings

In natural language applications, text is generally converted into discrete tokens before being processed by a transformer.

A simplified pipeline is:

$\text{Text}\to \text{Tokens}\to \text{Token IDs}\to \text{Embedding Vectors}\to \text{Transformer}$

Each token is represented by a numerical vector, allowing the neural network to process linguistic information mathematically.

Training

Transformer models are trained by adjusting their parameters to minimize a specified loss function.

For a model with parameters $\theta$, a simplified objective is:

${\theta }^{\ast }=\arg \underset{\theta }{min}ℒ(\theta )$

where $ℒ(\theta )$ represents the training loss.

For language modeling, a common objective is to predict a token based on contextual information.

A simplified autoregressive objective is:

$P(x_1,\dots ,x_n)=\prod _{t=1}^{n}P(x_t\mid x_1,\dots ,x_{t-1})$

where $x_t$ represents the token at position $t$.

The model is trained to assign high probability to appropriate next tokens.

Types of transformer models

Transformer architectures can be broadly categorized according to how their encoder and decoder components are used.

Encoder-only models

Encoder-only models use transformer encoders to produce contextual representations of input sequences.

They are commonly used for tasks such as:

• text classification;
• semantic representation;
• information retrieval;
• token classification;
• sentence similarity.

A simplified representation is:

$\text{Input}\to \text{Encoder}\to \text{Contextual Representation}$

Decoder-only models

Decoder-only models use transformer decoder blocks and are commonly used for autoregressive generation.

A simplified representation is:

$(x_1,\dots ,x_{t-1})\to \text{Decoder}\to P(x_t\mid x_1,\dots ,x_{t-1})$

where $x_t$ represents the predicted next token.

The model generates a sequence by repeatedly predicting the next token based on the tokens that have already been generated.

A simplified generation process is:

$x_1\to x_2\to \cdots \to x_t\to x_{t+1}$

Repeating this process allows the model to generate sequences within its supported context and generation limits.

Encoder–decoder models

Encoder–decoder models use both components.

A simplified representation is:

$\text{Input}\to \text{Encoder}\to \text{Decoder}\to \text{Output}$

This architecture is commonly associated with sequence-to-sequence tasks.

Causal attention

Decoder-based autoregressive transformers generally use causal or masked self-attention so that a token cannot directly attend to future tokens during training.

A simplified representation is:

$\text{Attention at Position }t\to \text{Positions }\le t$

This preserves the autoregressive property of next-token prediction.

Computational characteristics

For a sequence of length $n$, standard self-attention requires computing interactions between pairs of positions.

The attention matrix has approximately:

$n\times n=O(n^2)$

pairwise relationships.

Consequently, the computational and memory requirements of standard self-attention can increase substantially as sequence length grows.

Various transformer variants have been developed to reduce the computational cost associated with long sequences.

Applications

Transformers are used in a wide range of machine-learning applications, including:

• Natural language processing;
• machine translation;
• text generation;
• question answering;
• information retrieval;
• speech recognition;
• image classification;
• computer vision;
• multimodal learning;
• code generation;
• document processing;
• time-series modeling.

Their use has expanded beyond natural language because the attention mechanism can model relationships between elements in many types of structured data.

Advantages

Transformers have several important characteristics:

• attention can model relationships between distant elements of a sequence;
• training can be highly parallelizable compared with recurrent architectures;
• the architecture can be scaled to large numbers of parameters;
• the same general architecture can be adapted to different modalities and tasks;
• pretrained transformer models can be fine-tuned or adapted for downstream applications.

Limitations

Transformer models also have limitations.

These can include:

• high computational requirements;
• substantial memory requirements;
• increasing attention costs for long sequences;
• dependence on large training datasets for many applications;
• potential generation of inaccurate or fabricated information;
• sensitivity to training data and model configuration;
• substantial energy and infrastructure requirements for large-scale training and inference.

The limitations vary substantially among model architectures and applications.

Transformer and recurrent neural networks

Transformers differ from traditional recurrent neural networks (RNNs) in how sequence relationships are processed.

A simplified recurrent representation is:

$h_t=f(x_t,h_{t-1})$

where the current hidden state depends on the previous hidden state.

A simplified transformer representation is:

$h_t=f(x_1,\dots ,x_n)$

where the representation at a position can incorporate information from other positions through attention, subject to the model's attention structure.

Transformers therefore provide a different mechanism for modeling long-range relationships and enable greater parallelization during training.

Scaling

Transformer models can be scaled by increasing parameters, training data, computational resources, and other architectural dimensions.

A simplified relationship is:

$\text{Model Capability}\to f(\text{Parameters},\text{Data},\text{Compute},\text{Architecture})$

The relationship between scale and model performance is not universally linear, and practical performance depends on model architecture, data quality, optimization, evaluation methodology, and deployment conditions.

See also

• Artificial neural network
• Deep learning
• Attention mechanism
• Self-attention
• Multi-head attention
• Large language model
• Natural language processing
• Recurrent neural network
• Generative artificial intelligence
• Machine learning