The correct robots.txt for the site - why you need it, how to compose it and where to check
Currently, almost every site has a special file that serves to provide search engines with information on how to index the site correctly. This file is called robots.txt - in fact, it is a simple text file with instructions. It contains bans, permissions and other guidelines for search engines.
You can compose such a file yourself, you just need to know some general rules. Here's a example robots.txt file and look at it below:
User-agent: * # Directories Disallow: / core / Disallow: / custom / Disallow: / modules / Allow: /custom/modules/*.js Allow: /custom/modules/*.css Allow: /custom/modules/*.jpg # Files Disallow: /license.txt Disallow: /index.php # Url Host: https://example.com Sitemap: https://example.com/sitemap.xml
Let's take a quick look at all the main elements of the given file:
- the keyword User-agent indicates for which search engine agents the rules are prescribed - the value * means that the rules are prescribed for all search robots;
- comments are marked with # , followed by any text;
- the keyword Disallow means that the specified address is not indexed, you can disable entire sections of the site;
- the keyword Allow is used to allow indexing;
- the keyword Host is used to set the main domain of the site;
- the keyword Sitemap is used to set the path to address map .
Using the above keywords, you can compose your robots.txt file as needed. After drawing up the rules, you need to check the robots.txt file using special services that will show if there are any errors in it and offer corrections and recommendations if necessary. It is best to use utilities from large services, for example, Webmaster from Google or Yandex.
Thus, the article covered why robots.txt is needed , how to compose it correctly and where to check it.
- IT / Misc 08.07.21 How to make a free translation for a website without an API, translate documents in Google Translate
- IT / Misc 06.07.21 How to make a subscription button on a website, a subscriber base and automatic mailing
- Food / Misc 06.07.21 How to quickly cook delicious fried pies with potatoes and onions
- IT / Misc 04.07.21 Caching - create, load and reset. Where to store the cache, methods and types of caching
- IT / Database 03.07.21 Custom NoSQL - storing data in files and not only in a database. Storing settings, small data and caching files