Skip to main content
AI & LLM New

llms.txt & llms-full.txt Generator, Validator & Sitemap Importer

Comprehensive llms.txt and llms-full.txt generator for LLM search indexing (Perplexity, ChatGPT, Claude Search). Features visual section editor, sitemap.xml import, live BPE token counting, and llmstxt.org RFC validation.

Visual section and resource link manager (Title, URL, concise description, Required vs Optional)
Dual output tabs: Curated /llms.txt index and comprehensive /llms-full.txt markdown bundle
One-click sitemap.xml parser: Upload or paste sitemap XML to extract pages automatically
Live BPE token counter and context window percentage estimator (Claude 200k, GPT-4o 128k)
Strict llmstxt.org specification validator with error/warning diagnostic pills
Pre-loaded presets: SaaS Documentation, Developer Library, REST API Reference, and Portfolio
WebCraftKit Manifesto 100% Client-Side Engine

Air-Gapped Privacy & Zero-Latency Developer Utilities

Every cryptographic algorithm, schema transformer, color space converter, and binary extractor runs entirely in your browser RAM. Your tokens, API secrets, and source code are never sent to external servers.

Zero Server Telemetry
Sub-Millisecond Execution
70 Production Tools
Read Architecture Story →
Comprehensive Technical Manual

The Definitive Guide to llms.txt: Indexing Your Website for AI Search Engines

In-depth specifications, architectural mechanics, real-world code implementations, and industry best practices.

01

What is llms.txt and Why Does Your Website Need It?

When AI models (such as ChatGPT Search, Perplexity AI, Claude Search, and NotebookLM) search the web to answer user questions about your product, library, or company, they face heavy HTML boilerplate (scripts, navbars, ads, trackers) that dilutes context and wastes tokens.

`llms.txt` (specified at llmstxt.org) is the standardized markdown file hosted at `https://yourdomain.com/llms.txt` (similar to `robots.txt`) that provides AI search crawlers with a clean, curated markdown roadmap of your most important documentation and guides.

02

The llmstxt.org File Format Specification

A valid `llms.txt` file consists of structured markdown sections: 1. Top-level H1 title (`# Project Name`) 2. Summary Blockquote (`> Concise 1-2 sentence description of what the site is`) 3. Curated Sections (`## Core Documentation`, `## API Reference`) 4. Link list with markdown format: `- [Resource Title](https://domain.com/doc.md): Description of why this page matters.` 5. Optional "Optional" subsection for secondary resources (`### Optional`).

Implementation Example
# WebCraftKit

> Ultra-fast, 100% browser-native developer tools and technical engineering guides with zero server latency.

## Core Documentation
- [Developer Utilities Overview](https://webcraftkit.com/tools): Full index of 57 client-side developer, design, and AI tools.
- [JWT Security Guide](https://webcraftkit.com/blog/jwt-security-best-practices): Production guide to JSON Web Token architecture and signature verification.
- [Privacy Architecture](https://webcraftkit.com/privacy): Detailed breakdown of zero-server data processing and Web Crypto API.

## Optional
- [About Us](https://webcraftkit.com/about): Mission, founder background (TitanByte), and philosophy.
03

Difference Between /llms.txt and /llms-full.txt

- **`/llms.txt`:** A lightweight, token-efficient index (typically 500 to 2,000 tokens) pointing to clean markdown URLs. - **`/llms-full.txt`:** A single, concatenated bundle containing the entire raw text of all documentation pages, ideal for feeding an entire library into an LLM context window in a single request.

04

Importing from sitemap.xml and Token Budgeting

Our studio features a browser-native `sitemap.xml` parser. Paste or upload your XML sitemap, select the key documentation URLs, and generate a validated `llms.txt` file in seconds. The live token counter ensures your index stays lean and fits within the lowest LLM token budgets.

Knowledge Base & Clarifications

Frequently Asked Questions: llms.txt Generator

Got questions about how llms.txt Generator operates, client-side cryptographic safety, or performance limits? Explore common answers below.

Complementary Utilities
View all in AI & LLM →