Why is token overlap essential when splitting text for RAG?

Token overlap prevents semantic fracture. If a key fact or sentence is split across two chunks without overlap, neither chunk may contain sufficient context to match the user query vector in embedding space. Overlap ensures that boundary concepts exist intact within at least one chunk.

What is the ideal chunk size for OpenAI text-embedding-3-small or large?

While text-embedding-3 supports up to 8,191 tokens per input, optimal retrieval accuracy is typically achieved with chunk sizes of 400 to 800 tokens with 10–15% overlap. Smaller chunks provide higher semantic specificity for vector similarity search.

How does Recursive Character Splitting differ from basic character splitting?

Basic character splitting cuts text strictly every N characters, frequently breaking words and sentences mid-syllable. Recursive Character Splitting attempts to divide text first at paragraph breaks (\n\n), then sentence breaks (\n), and word breaks (spaces), preserving semantic integrity.

How should code and technical documentation be chunked differently than prose?

Code and markdown require syntax-aware splitting that respects function definitions, code blocks (```), and header tags (#). Splitting inside a code block destroys syntax validity, so specialized separators (e.g., class and function definitions) are preferred.

Are my uploaded documents or knowledge base files stored on your servers?

No. WebCraftKit processes all documents 100% locally inside your browser runtime. Zero text, files, or vector embeddings are transmitted over the network or stored on any server.

AI & LLM New

RAG Document Text Chunker & Token Overlap Splitter

Split documents, Markdown, code, and text for Retrieval-Augmented Generation (RAG) using recursive character, token-based, and semantic boundary chunking with configurable overlap.

Multiple Chunking Strategies: Recursive Character Splitting, Token-Length Windowing, Paragraph/Sentence Boundary Splitting, and Markdown Header hierarchy segmentation

Configurable Overlap Percentage: Set chunk overlap (0% to 50% / 10–200 tokens) to maintain contextual coherence across chunk seams and eliminate split-context retrieval failures

Interactive Visual Chunk Inspector: Inspect individual chunks color-coded by boundary, verify token counts per chunk, and review overlapping token segments

Code & Markdown Syntax Preservation: Intelligent splitting logic that protects code blocks, Markdown headers, table rows, and list items from disruptive mid-block truncation

Vector DB Ingestion Payloads: Export chunk arrays with calculated metadata (chunk_id, char_start, char_end, token_count, section_header) as JSON, CSV, or Python dictionaries ready for Pinecone, Qdrant, Chroma, and Milvus

100% In-Browser Privacy: Safely chunk proprietary enterprise documents, internal PDFs, and private codebases without uploading sensitive text to external servers

The Complete Engineering Guide to RAG Document Chunking & Text Splitting

In-depth specifications, architectural mechanics, real-world code implementations, and industry best practices.

The Critical Role of Chunking in Retrieval-Augmented Generation (RAG)

In Retrieval-Augmented Generation (RAG) architectures, Large Language Models query external vector databases to retrieve relevant context before generating an answer. Chunking is the foundational process of breaking large documents (PDFs, documentation, code repositories, knowledge base articles) into smaller, semantically coherent passages. If chunks are too large, vector embeddings dilute specific facts, retrieval precision plummets, and prompt context windows get exhausted. If chunks are too small, crucial contextual nuance is severed, leaving the LLM unable to synthesize complete answers. Selecting the appropriate chunking strategy and overlap is the single highest-leverage optimization for RAG retrieval quality.

Implementation Example

// RAG Ingestion Pipeline:
// Raw Source File -> Document Cleaner -> Chunking & Overlap Splitter -> Vector Embedding Model -> Vector Database (Pinecone / Qdrant)

Algorithmic Breakdown: Recursive Character vs Token vs Markdown Splitting

• Recursive Character Text Splitting: Iteratively attempts to split text using a hierarchical list of separators—starting with double newlines (\n\n for paragraphs), single newlines (\n for sentences), spaces ( ) for words, and finally individual characters. This preserves natural semantic boundaries whenever possible.
• Token-Based Windowing: Measures exact subword token counts using models like tiktoken (cl100k_base or o200k_base), ensuring no single chunk ever exceeds the hard token limit of embedding models (e.g., 512 or 8,192 tokens).
• Markdown Header Hierarchical Splitting: Splits text along # H1, ## H2, and ### H3 structural boundaries, carrying header breadcrumb metadata into each chunk to maintain document hierarchy in vector search.
• Token Overlap Mechanics: Retains a sliding window of trailing tokens from chunk N as the leading prefix of chunk N+1, preventing sentences from being sliced in half across chunk seams.

Implementation Example

// Standard recursive separator hierarchy in LangChain / LlamaIndex
const defaultSeparators = [
  "\n\n", // Paragraph breaks (highest priority)
  "\n",   // Line / sentence breaks
  ". ",   // Sentence terminators
  "! ", "? ",
  "; ",
  " ",    // Word boundaries
  ""      // Character fallback (lowest priority)
];

Step-by-Step Tutorial: Preparing a Knowledge Base for Vector Ingestion

Step 1: Paste or upload your raw source document (Markdown, plain text, code, or transcription). Step 2: Select your Target Embedding Model (e.g., OpenAI text-embedding-3-small with 512-token target, or BGE / Cohere Embed v3). Step 3: Configure Chunk Size and Overlap—Set chunk target to 400–500 tokens with a 15% (60 token) overlap for standard Q&A documentation. Step 4: Inspect Visual Chunks—Review chunk length distribution, check that code blocks remain intact, and verify that overlap boundaries capture complete thoughts. Step 5: Export Vector Payloads—Download the parsed chunks formatted with full metadata (id, content, token_count, metadata: { source, chunk_index }) ready for upserting into Pinecone, Qdrant, ChromaDB, or pgvector.

Implementation Example

// Sample Exported Vector DB Ingestion Payload (JSON)
[
  {
    "id": "doc_001_chunk_0",
    "values": [/* 1536-dim embedding vector */],
    "metadata": {
      "text": "Retrieval-Augmented Generation (RAG) optimizes LLM responses...",
      "token_count": 412,
      "source": "knowledge-base-v2.md",
      "chunk_index": 0,
      "has_overlap": true
    }
  }
]

Implementing Production Text Splitters in Python & TypeScript

Production RAG pipelines implement text splitting using LangChain, LlamaIndex, or lightweight standalone libraries. Below are production implementations in Python using LangChain's RecursiveCharacterTextSplitter and in TypeScript/Node.js using @langchain/textsplitters.

Implementation Example

// TypeScript: Production Text Splitting with @langchain/textsplitters
import { RecursiveCharacterTextSplitter } from '@langchain/textsplitters';

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 500,
  chunkOverlap: 50,
  separators: ['\n\n', '\n', ' ', ''],
});

export async function chunkDocument(rawText: string) {
  const docs = await splitter.createDocuments([rawText]);
  return docs.map((doc, i) => ({
    chunkId: `chunk_${i}`,
    content: doc.pageContent,
    charLength: doc.pageContent.length,
  }));
}

// Python: LangChain Recursive Character Splitter
// from langchain_text_splitters import RecursiveCharacterTextSplitter
// splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
// chunks = splitter.split_text(raw_text)

Chunk Size Benchmarks, Retrieval Tradeoffs & Privacy Assurance

• Short-Form Retrieval (256–512 tokens): Optimal for precision question-answering, entity lookups, and FAQ matching with high top-k reranking.
• Long-Form Retrieval (800–1500 tokens): Optimal for narrative summarization, complex legal/financial analysis, and multi-step reasoning where extensive context is necessary.
• Overlap Rule of Thumb: Maintain 10% to 20% overlap. Zero overlap causes up to 15% drop in retrieval accuracy for questions answering cross-boundary facts.
• Total Privacy: WebCraftKit processes all document parsing and token calculations locally in your browser memory. Confidential proprietary documents, HR policies, and intellectual property never touch external infrastructure.

Implementation Example

// Recommended Parameter Matrix:
// Task                     | Chunk Size   | Overlap | Strategy
// Precision QA & Search    | 256-512 tok  | 15%     | Recursive Character
// Codebase Search          | 400-800 tok  | 10%     | AST / Syntax-Aware
// Legal & Document Summary | 1000-1500 tok| 20%     | Markdown / Sectional

Knowledge Base & Clarifications