Natural language processing

Natural language processing (NLP) is a field of artificial intelligence and Computational linguistics concerned with enabling computers to process, analyze, interpret, and generate human language. NLP combines methods from computer science, linguistics, statistics, and Machine learning to process language in written, spoken, and other digital forms.

Natural language processing is used in applications including machine translation, speech recognition, information retrieval, text classification, sentiment analysis, document analysis, question answering, conversational systems, and generative AI.

Modern NLP systems commonly use Deep learning and neural network architectures, including transformers and large language models. These systems can process large quantities of textual and linguistic data and perform multiple language-related tasks.

History

Natural language processing developed from research in computational linguistics, formal language theory, information retrieval, and statistical language modelling.

Early NLP systems relied heavily on manually constructed linguistic rules and symbolic representations. Statistical methods subsequently became increasingly important as larger language datasets and greater computing resources became available.

The development of neural networks and deep learning substantially changed NLP by enabling models to learn linguistic representations directly from large datasets. Transformer-based architectures subsequently enabled major advances in language understanding and generation.

Objectives

The principal objectives of natural language processing are to:

enable computers to process human language;
extract information from textual and spoken data;
interpret linguistic meaning;
classify and organize language data;
translate between languages;
generate natural-language content;
support human-computer interaction.

Major tasks

Natural language processing encompasses a wide range of tasks, including:

Tokenization;
Part-of-speech tagging;
Named-entity recognition;
Sentiment analysis;
text classification;
information extraction;
information retrieval;
machine translation;
speech recognition;
question answering;
text summarization;
natural-language generation.

Tokenization

Tokenization is the process of dividing text into smaller units, such as words, subwords, or tokens.

Tokenization provides a representation that can be processed by computational language models.

Part-of-speech tagging

Part-of-speech tagging assigns grammatical categories such as noun, verb, adjective, or adverb to words or tokens within a sentence.

Named-entity recognition

Named-entity recognition identifies entities within text, such as people, organizations, locations, dates, and other predefined categories.

Sentiment analysis

Sentiment analysis evaluates text to determine the expressed sentiment, opinion, or attitude.

It is commonly used for analyzing customer feedback, social media content, reviews, and other textual sources.

Information extraction

Information extraction identifies structured information from unstructured or semi-structured language.

Applications may include extracting names, dates, relationships, events, financial information, or other domain-specific data.

Machine translation

Machine translation uses computational systems to translate text or speech between languages.

Modern machine translation systems commonly use neural network architectures.

Text summarization

Text summarization generates a shorter representation of a document or collection of documents while attempting to preserve important information.

Summarization may be extractive or abstractive.

Question answering

Question answering systems process questions expressed in natural language and generate or retrieve relevant answers.

They may use information retrieval, language models, knowledge bases, or combinations of these technologies.

Methods

NLP systems may use several methodological approaches, including:

rule-based methods;
statistical methods;
machine learning;
deep learning;
neural language models;
transformer-based models.

Different methods may be combined depending on the application and available data.

Machine learning

Machine learning is widely used in modern NLP.

Machine-learning systems can learn relationships and patterns from linguistic datasets rather than relying exclusively on manually specified rules.

Deep learning

Deep learning has become a major approach in NLP because neural networks can learn complex representations of language from large datasets.

Deep-learning architectures have been applied to language classification, translation, speech recognition, text generation, and other NLP tasks.

Transformers

The transformer architecture is widely used in modern NLP systems.

Transformers use mechanisms such as self-attention to model relationships between tokens within a sequence. They form the basis of many contemporary language models.

Large language models

Large language models are a major class of modern NLP systems.

LLMs are trained on large datasets and can perform tasks such as text generation, summarization, translation, question answering, classification, and information extraction.

Applications

Natural language processing is used in:

search engines;
virtual assistants;
chatbots;
machine translation;
document processing;
information retrieval;
customer service;
content analysis;
sentiment analysis;
healthcare information systems;
financial analysis;
legal document analysis;
academic research;
business intelligence.

Relationship with artificial intelligence

Artificial intelligence is the broader field concerned with developing computational systems capable of performing tasks associated with human intelligence.

Natural language processing is a major subfield of artificial intelligence focused specifically on human language.

Relationship with computational linguistics

Computational linguistics applies computational methods to the study of human language.

NLP and computational linguistics overlap substantially, although computational linguistics traditionally places greater emphasis on linguistic theory and formal analysis, while NLP commonly focuses on practical computational applications.

Relationship with generative artificial intelligence

Generative artificial intelligence includes systems capable of generating new content.

NLP provides many of the techniques used to process and generate natural-language content, while modern generative AI systems frequently use NLP-based architectures such as large language models.

Limitations

Natural language processing systems may have difficulty with:

ambiguity;
context-dependent meaning;
sarcasm and irony;
idiomatic expressions;
multilingual variation;
low-resource languages;
domain-specific terminology;
factual verification.

Modern systems may also reproduce biases present in training data or generate inaccurate information.

Importance

Natural language processing is a fundamental field within modern artificial intelligence because it enables computers to interact with and analyze human language at scale. Its development has contributed to major advances in search, translation, information retrieval, conversational systems, document analysis, and generative AI, making language processing an important component of modern information and knowledge systems.