The correct robots.txt for the site - why you need it, how to compose it and where to check
Currently, almost every site has a special file that serves to provide search engines with information on how to index the site correctly. This file is called robots.txt - in fact, it is a simple text file with instructions. It contains bans, permissions and other guidelines for search engines.
You can compose such a file yourself, you just need to know some general rules. Here's a example robots.txt file and look at it below:
Disallow: / core /
Disallow: / custom /
Disallow: / modules /
Let's take a quick look at all the main elements of the given file:
- the keyword User-agent indicates for which search engine agents the rules are prescribed - the value * means that the rules are prescribed for all search robots;
- comments are marked with # , followed by any text;
- the keyword Disallow means that the specified address is not indexed, you can disable entire sections of the site;
- the keyword Allow is used to allow indexing;
- the keyword Host is used to set the main domain of the site;
- the keyword Sitemap is used to set the path to address map .
Using the above keywords, you can compose your robots.txt file as needed. After drawing up the rules, you need to check the robots.txt file using special services that will show if there are any errors in it and offer corrections and recommendations if necessary. It is best to use utilities from large services, for example, Webmaster from Google or Yandex.
Thus, the article covered why robots.txt is needed , how to compose it correctly and where to check it.
- 09.11.23IT / Database Errors when migrating from MySQL 5.6 to 5.7 and how to fix them - database dump import failed with an error or INSERT does not work. Disabling STRICT_TRANS_TABLES strict mode or using IGNORE
- 09.07.22IT / Misc Convert office files DOC, DOCX, DOCM, RTF to DOCX, DOCM, DOC, RTF, PDF, HTML, XML, TXT formats without loss and markup changes
- 07.07.22IT / Safety How to protect PHP, JS, HTML, CSS source code - obfuscation, minification, compression and encryption
- 06.07.22IT / Safety Connection not secure, problem with Lets Encrypt - how to fix expired 09/30/2021 DST Root CA X3, remove it manually and install ISRG Root X1. Example on MS Windows 7
- 08.07.21IT / Misc How to make a free translation for a website without an API, translate documents in Google Translate