Generate a robots.txt file for your website. Control how search engine crawlers access your site with allow, disallow rules and sitemap references.
The robots.txt file is a text file placed at the root of your website that instructs search engine crawlers which pages or sections they can or cannot access. It is the first file crawlers look for when visiting your site, making it a critical component of your SEO strategy.
robots.txt = User-agent: [bot] \n Allow: [path] \n Disallow: [path] \n Sitemap: [url]A robots.txt file consists of one or more groups of directives. Each group starts with a User-agent line specifying which crawler the rules apply to, followed by Allow and/or Disallow directives, and optionally a Sitemap URL pointing to your XML sitemap.
| Input | Output |
|---|---|
| User-agent: *, Disallow: /admin/, Sitemap: https://example.com/sitemap.xml | User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml |
| User-agent: Googlebot, Allow: /content/, Disallow: /private/ | User-agent: Googlebot Allow: /content/ Disallow: /private/ |
| User-agent: *, Disallow: /tmp/, Disallow: /search/ | User-agent: * Disallow: /tmp/ Disallow: /search/ |