How robots.txt should be for a website?
A file named robots.txt should be created in the root folder of the site. The format of this file should be saved as without-bom. The contents of this file should be as follows.
The following rules should be observed while creating this file.
1. If all pages are wanted to be indexed, which is often the case. Only the Allow tag should be used in the file. The file should be as follows.
User-agent: *
Allow: /
Disallow: /404
Sitemap: https://domain.com/sitemap.xml
2. In this file, the link of the 404 page should be specified as Disallow directly.
3. In this file, the main folder where sitemap.xml is installed should be added directly with the domain structure to be used. For example, if we access the domain with https://sitename.com, the site should be specified like this, if we access it as https://www.sitename.com or if we want it to be indexed in this way, it should be added using www.
Write a comment