Analyzing the Presence and Usage of llms.txt Files Across 137,000 Websites: Key Insights
In the rapidly evolving landscape of internet sites and AI integration, the role and visibility of llms.txt files have become a point of discussion among developers, SEO professionals, and AI researchers alike. Despite the growing chatter, concrete data on how these files are accessed and utilized remains limited. To shed light on this topic, we conducted a comprehensive analysis of server logs and live traffic from a sizable sample of 137,000 domains, employing tools such as Ahrefs Web Analytics and Bot Analytics.
Our findings offer a data-driven perspective on the prevalence and engagement with llms.txt files across a diverse range of websites.
Key Findings
-
Prevalence of llms.txt Files
Approximately 28% of the analyzed domains hosting Ahrefs Web Analytics have publicly accessible llms.txt files. This indicates that a significant minority of websites maintain these files, potentially as part of their AI or bot management strategies. -
Traffic to llms.txt Files
Astonishingly, 97% of these files received no traffic during May 2026. In other words, nearly all of the files weren’t actively fetched by any user agent or bot during this period, suggesting limited or no engagement. -
Bot-Centric Access
Among the requests that did reach the llms.txt files, an overwhelming 96% originated from automated bots. This underscores that, when accessed, the files are predominantly targeted by machine agents rather than human visitors. -
AI Tools and Search Bots
Of the fetches involving llms.txt files, 19.5% came from recognized AI tools and search engines. Notably, GPTBot leads the charge, followed by Claude-Code, edging out other AI search and assistant bots. This pattern highlights ongoing AI indexing efforts regarding these files. -
Self-Study and Research
Approximately 12% of requests originate from industry-specific tools—such as GEO/AEO scanners, llms.txt verification tools, and academic researchers—indicating targeted efforts to examine or validate these files. -
Absence of AI Bot Requests for Non-Existent Files
Interestingly, no AI bots attempted to fetch llms.txt files that do not exist on a site. This suggests that AI agents are not probing for non-configured or missing files, possibly reflecting their reliance on existing configurations or indications that these files are present. -
Influence of Performance Audits
The Chrome Lighthouse llms.txt audit accounted for roughly 0.1% of fetches, signifying that some website audits or performance checks also query these files, albeit infrequently.
Implications and Takeaways
These insights reveal that while a notable portion of websites host llms.txt files, actual traffic to these files—particularly from AI agents—is minimal and largely limited to specific bots and research tools. The overwhelming majority of these files remain untouched in live traffic, raising questions about their practical impact and the perception of their existence.
Furthermore, the dominant access by AI and bot entities underscores the ongoing interest from search engines and AI systems in understanding or indexing site configurations related to AI interactions.
For website administrators and AI developers, these findings emphasize the importance of thoughtful configuration and awareness of how such files are being accessed and utilized. As the ecosystem continues to evolve, ongoing monitoring and analysis will be critical to optimize AI integration and maintain transparency.
Note: This analysis is based on observed server logs and live traffic data collected from a diverse set of websites in May 2026, offering a snapshot of current trends in the usage of llms.txt files across the web.
