Artificial neural network

Artificial neural network (ANN) is a computational model composed of interconnected processing units, commonly called neurons, that is designed to learn patterns and relationships from data. Artificial neural networks are a major component of Machine learning and form the foundation of many Deep learning systems.

An artificial neural network typically consists of an input layer, one or more hidden layers, and an output layer. Each connection between computational units has an associated weight, and the network learns by adjusting these weights during training to reduce the difference between its predictions and the desired outputs.

Artificial neural networks are used in artificial intelligence applications including classification, regression, pattern recognition, computer vision, speech recognition, Natural language processing, forecasting, anomaly detection, and generative AI.

History

Artificial neural networks were inspired in part by research into biological neural systems and the computational properties of neurons. Early neural-network research focused on mathematical models of artificial neurons and methods for learning relationships between inputs and outputs.

The development of multilayer networks, backpropagation, improved optimization techniques, larger datasets, and more powerful computing hardware contributed to the expansion of neural networks within machine learning.

The subsequent development of Deep learning enabled neural networks containing many layers to learn increasingly complex representations from large datasets.

Structure

An artificial neural network generally consists of interconnected layers of computational units.

A basic network may contain:

an input layer;
one or more hidden layers;
an output layer.

The input layer receives data, hidden layers transform the information, and the output layer produces the model's prediction or generated output.

Neuron

A computational unit within a neural network receives inputs, applies associated weights, and produces an output.

A simplified representation of a neuron's weighted input is:

$z = \sum_{i = 1}^{n} w_{i} x_{i} + b$

where:

x_i = input values;
w_i = associated weights;
b = bias;
z = weighted input.

An activation function is then applied to determine the neuron's output.

Activation function

An activation function transforms the weighted input of a neuron into an output that is passed to subsequent layers.

Common activation functions include:

sigmoid;
hyperbolic tangent;
rectified linear unit (ReLU);
softmax.

Nonlinear activation functions allow neural networks to model relationships that cannot be represented effectively by a purely linear transformation.

Weights and biases

Weights determine the strength of connections between computational units.

Biases provide additional parameters that allow the network to shift the output of individual neurons.

During training, the network adjusts its weights and biases to improve its performance on the specified task.

Training

Training an artificial neural network involves presenting data to the model and adjusting its parameters according to a defined learning objective.

A simplified optimization problem can be represented as:

$θ^{*} = \arg min_{θ} L (θ)$

where:

θ = model parameters;
L(θ) = loss function;
θ* = parameter values that minimize the loss under the training procedure.

Training commonly uses backpropagation together with optimization algorithms such as gradient descent.

Backpropagation

Backpropagation is an algorithm used to calculate how changes in network parameters affect the model's loss.

The resulting gradients are used by an optimization algorithm to update the network's weights and biases.

Backpropagation is a fundamental training technique for many neural networks.

Major architectures

Artificial neural networks include several architectures designed for different types of data and tasks.

Common architectures include:

Feedforward neural networks

Feedforward neural networks process information in one primary direction, from the input layer through intermediate layers to the output layer.

They are commonly used for classification and regression tasks.

Convolutional neural networks

Convolutional neural networks (CNNs) are specialized for processing spatial data and are widely used in computer vision.

They can identify local patterns such as edges, shapes, and other visual features.

Recurrent neural networks

Recurrent neural networks (RNNs) are designed to process sequential data by incorporating information from previous elements of a sequence.

They have been used in language processing, speech recognition, and time-series analysis.

Transformers

The transformer architecture uses self-attention mechanisms to model relationships between elements of a sequence.

Transformers have become a major architecture in modern natural language processing and are widely used in large language models and multimodal AI systems.

Autoencoders

Autoencoders are neural networks that learn to represent input data in a compressed or structured form and subsequently reconstruct the input.

They can be used for representation learning, dimensionality reduction, anomaly detection, and generative applications.

Applications

Artificial neural networks are used in:

Computer vision;
Natural language processing;
speech recognition;
machine translation;
image classification;
forecasting;
recommendation systems;
fraud detection;
cybersecurity;
scientific research;
robotics;
Generative artificial intelligence.

Relationship with machine learning

Machine learning is the broader field concerned with computational methods that learn patterns from data.

Artificial neural networks are one class of machine-learning models and are particularly important in modern machine learning because of their ability to learn complex nonlinear relationships.

Relationship with deep learning

Deep learning is a branch of machine learning that uses neural networks with multiple layers to learn hierarchical representations.

Artificial neural networks form the underlying computational structure of deep-learning systems.

Relationship with artificial intelligence

Artificial intelligence is the broader field concerned with computational systems capable of performing tasks associated with human intelligence.

Artificial neural networks provide an important computational approach for developing modern AI systems.

Relationship with natural language processing

Natural language processing uses computational techniques to process and generate human language.

Neural networks are widely used in modern NLP systems, particularly transformer-based architectures and large language models.

Relationship with generative artificial intelligence

Generative artificial intelligence uses neural-network-based models to generate new content such as text, images, audio, video, and software code.

Different neural-network architectures are used for different types of generative tasks.

Advantages

Artificial neural networks can model complex nonlinear relationships, learn representations directly from data, and process high-dimensional datasets.

They can also be adapted to a wide range of tasks by changing their architecture, training data, and learning objective.

Limitations

Artificial neural networks may require substantial amounts of:

training data;
computational resources;
training time;
storage.

Other limitations include limited interpretability, sensitivity to training data, potential bias, susceptibility to adversarial inputs, and difficulty determining why complex models produce particular outputs.

Poorly designed or insufficiently trained networks may also overfit their training data and perform poorly on new data.

Importance

Artificial neural networks are a foundational technology in modern machine learning and artificial intelligence. Their ability to learn complex representations from data has enabled major advances in deep learning, computer vision, natural language processing, speech recognition, generative AI, and other computational fields.