Generate a valid robots.txt file for your WordPress website in seconds. Tick what to block, add your sitemap URL and copy the result. Free, no signup required.
A robots.txt file sits at the root of your website and tells search engine bots which pages they are and are not allowed to crawl. It is one of the most important technical SEO files on your site. A correctly configured robots.txt prevents Google from wasting crawl budget on pages that should not be indexed — like WordPress admin, search results, feed URLs and WooCommerce checkout pages.
Upload your generated robots.txt file to your website root directory (the same folder as your homepage) via FTP, your hosting file manager, or your WordPress SEO plugin settings.
A single wrong rule in robots.txt can accidentally block Google from crawling an entire site, one of the most common and easily missed causes of a site vanishing from search results. Generating a clean, correct file removes a surprisingly common source of technical SEO disasters.
Yes. A rule like Disallow: / blocks crawlers from every page on the site. This is one of the most common technical mistakes we see, often introduced accidentally during a site migration or redesign.
Not always. It stops crawling, but a blocked page can still appear in search results without a description if other sites link to it. To fully remove a page, use a noindex meta tag instead, on a page Google is allowed to crawl.
It is good practice. Adding a Sitemap: line pointing to your XML sitemap makes it easier for search engines to discover all your pages efficiently.
We use cookies for essential site functionality and, with your consent, for analytics and ad personalisation. See our Privacy Policy to learn more.