Chunking
The process of splitting large documents into smaller, overlapping segments optimized for retrieval and embedding.
Chunking breaks lengthy documents into pieces that fit within an embedding model's token limit while preserving enough context for each chunk to be independently meaningful. Common strategies include fixed-size windows, sentence-boundary splitting, and heading-aware segmentation.
The choice of chunk size and overlap directly impacts retrieval quality. Chunks that are too small lose context; chunks that are too large dilute relevance scores. For compliance documents, heading-aware chunking preserves clause boundaries and section references.
More ai/ml Terms
Retrieval-Augmented Generation (RAG)
An AI architecture that combines information retrieval with text generation to produce answers grounded in source documents.
Vector Embedding
A numerical representation of text as a high-dimensional vector, enabling semantic similarity comparisons between passages.
BM25
A probabilistic keyword-ranking algorithm that scores documents by term frequency and inverse document frequency.
Hallucination
When an AI model generates plausible-sounding but factually incorrect or fabricated information.
Large Language Model (LLM)
A neural network trained on massive text corpora that can understand and generate human language.
Fine-Tuning
The process of further training a pre-trained model on domain-specific data to improve its performance on targeted tasks.
Analyze Documents Related to Chunking
Upload any document and get AI-powered analysis with verifiable citations.
Start Free