The correct robots.txt for the site - why you need it, how to compose it and where to check

27.08.18 IT / SEO 1971

Currently, almost every site has a special file that serves to provide search engines with information on how to index the site correctly. This file is called robots.txt - in fact, it is a simple text file with instructions. It contains bans, permissions and other guidelines for search engines.

You can compose such a file yourself, you just need to know some general rules. Here's a example robots.txt file and look at it below:

  User-agent: *

# Directories

Disallow: / core /
Disallow: / custom /
Disallow: / modules /

Allow: /custom/modules/*.js
Allow: /custom/modules/*.css
Allow: /custom/modules/*.jpg

# Files

Disallow: /license.txt
Disallow: /index.php

# Url

Host: https://example.com
Sitemap: https://example.com/sitemap.xml
 

Let's take a quick look at all the main elements of the given file:

• the keyword User-agent indicates for which search engine agents the rules are prescribed - the value * means that the rules are prescribed for all search robots;
• comments are marked with # , followed by any text;
• the keyword Disallow means that the specified address is not indexed, you can disable entire sections of the site;
• the keyword Allow is used to allow indexing;
• the keyword Host is used to set the main domain of the site;
• the keyword Sitemap is used to set the path to address map .

Using the above keywords, you can compose your robots.txt file as needed. After drawing up the rules, you need to check the robots.txt file using special services that will show if there are any errors in it and offer corrections and recommendations if necessary. It is best to use utilities from large services, for example, Webmaster from Google or Yandex.

Thus, the article covered why robots.txt is needed , how to compose it correctly and where to check it.