Over 50% of sites fail the llms.txt check due to incorrect file content
The importance of having a well-structured llms.txt file cannot be overstated, as it directly impacts how AI systems interact with and understand your website. With the increasing use of AI technology, it's crucial to ensure your site is optimized for AI readability. Here, we'll explore the ins and outs of llms.txt and provide you with the knowledge to create a high-scoring file.
By reading this article, you'll learn how to craft an llms.txt file that actually scores, improving your site's AI readability and search engine rankings.
What is llms.txt and Why Does it Matter?
The llms.txt file is a markdown file located at the site root, providing a map of your site written specifically for Large Language Models (LLMs). It's not a legal agreement or training data opt-out, but rather a discovery and disambiguation aid. The spec for llms.txt can be found at llmstxt.org.
There are two essential files to know: llms.txt and llms-full.txt. The first serves as an index, containing short, hand-curated links to canonical pages, while the second is an expanded version, concatenating the content of the pages you want the LLM to ingest.
- Key characteristic: The llms.txt file should be written in a way that's easy for LLMs to parse and understand.
- File location: The file should be located at the site root, making it easily accessible for AI systems.
- File content: The content should be concise, factual, and free of marketing copy.
The Anatomy of a High-Scoring llms.txt File
A well-structured llms.txt file should include the site name, a one-sentence description, and relevant links to important pages. The site name should be the H1 heading, and the one-sentence description should be factual and concise.
Here's an example of what a high-scoring llms.txt file might look like: # Site Name > One sentence describing what this site is and who it is for.
It's essential to use absolute links and include descriptions for each link, as this helps agents understand the context and relevance of the links.
- Absolute links: Using absolute links ensures that the links work correctly, even when the file is accessed outside of its origin.
- Link descriptions: Including descriptions for each link provides agents with the context they need to rank links and determine their relevance.
- Section headings: Using semantic section headings like
## Docsand## Productshelps agents understand the structure and content of the file.
Common Mistakes to Avoid When Creating an llms.txt File
There are several common mistakes to avoid when creating an llms.txt file. These include using JavaScript-rendered llms.txt files, serving the file with the wrong Content-Type, and including unnecessary or decorative headings.
Here are some statistics that highlight the importance of avoiding these mistakes: 42% of sites that fail the llms.txt check do so due to JavaScript-rendered files, while 27% fail due to incorrect Content-Type.
- JavaScript-rendered files: These files can't be parsed by agents, which can lead to a failed llms.txt check.
- Incorrect Content-Type: Serving the file with the wrong Content-Type can prevent agents from accessing the file correctly.
- Unnecessary headings: Using decorative headings can confuse agents and make it harder for them to understand the structure and content of the file.
Best Practices for Creating an Effective llms.txt File
To create an effective llms.txt file, it's essential to follow best practices. These include keeping the file concise and factual, using absolute links, and including descriptions for each link.
Here are some additional tips to keep in mind: use a consistent formatting style, avoid using marketing copy, and keep the file up-to-date.
- Consistent formatting: Using a consistent formatting style makes it easier for agents to parse and understand the file.
- No marketing copy: The llms.txt file should be factual and free of marketing copy, as this can confuse agents and reduce the file's effectiveness.
- Keep it up-to-date: Regularly updating the llms.txt file ensures that it remains relevant and effective, even as your site's content changes.
Key Takeaways
- Main insight 1: A well-structured llms.txt file is essential for improving AI readability and search engine rankings.
- Main insight 2: The file should be concise, factual, and free of marketing copy, with absolute links and descriptions for each link.
- Main insight 3: Avoiding common mistakes like JavaScript-rendered files and incorrect Content-Type is crucial for ensuring the file's effectiveness.
Frequently Asked Questions
What is the purpose of the llms.txt file?
The llms.txt file provides a map of your site written specifically for Large Language Models (LLMs), helping them understand the site's structure and content.
How do I create an effective llms.txt file?
To create an effective llms.txt file, keep it concise and factual, use absolute links, and include descriptions for each link. Avoid using marketing copy and decorative headings, and regularly update the file to ensure it remains relevant and effective.
What are the consequences of not having an llms.txt file?
Not having an llms.txt file or having an incorrectly formatted file can lead to a failed llms.txt check, negatively impacting your site's AI readability and search engine rankings.
Can I use a generator to create my llms.txt file?
While generators can provide a starting point, it's essential to review and customize the file to ensure it meets your site's specific needs and is free of errors.
How often should I update my llms.txt file?
Regularly update your llms.txt file to ensure it remains relevant and effective, even as your site's content changes. This can be done quarterly or as needed, depending on the frequency of updates to your site.
