GPT Crawler is an open-source tool from Builder.io for crawling websites and turning their content into structured data that can be used with custom GPTs and other retrieval-based AI applications. Users provide a starting URL and matching rules, and the crawler visits relevant pages, extracts readable content, and produces an output dataset.

The project is designed to simplify the collection step for website-grounded assistants, documentation bots, and knowledge bases. Configuration controls which pages are included, how crawling proceeds, and where the resulting content is written, allowing developers to create focused corpora instead of manually copying pages one by one.

GPT Crawler is most useful when the source knowledge already exists on a public or accessible website and needs to be packaged for an LLM workflow. It is a lightweight ingestion utility rather than a complete RAG platform, making it easy to combine with the model, vector store, or assistant stack of the user's choice.