commoncrawl/llms.txt

llms.txt files extracted from the Common Crawl corpus This dataset contains llms.txt files extracted from the Common Crawl corpus. Specifically, we extracted response records matching */llms.txt and */llms-full.txt with status 200 and MIME-type text/plain or text/markdown. What is llms.txt? The /llms.txt file: A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time. Our own analysis of this dataset is available… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/llms.txt.

<iframe src="https://silytics.si/embed/item/fd24198e-a9c7-4579-a9a8-5bfe2fd035ef" width="420" height="190" style="border:0" loading="lazy" title="Silytics entry"></iframe>

Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.