Robots.txt article

Tips and Tricks
1

This is not Blocs specific, but nevertheless relevant to all Blocs users. I found this article on best practices with robots.txt quite useful, including the idea that they should list the sitemap url, along with differences between using robots.txt vs robot meta tags. Robots.txt Vs Meta Robots Tag: Which is Best? - In Front Digital

3 Likes
2

@Flashman thanks for making us aware of this important information.

3

Here is a good article on testing your robot.txt

1 Like
4

So, in what circumstances would you NOT want robots to crawl your site?

5

I do it all the time when preparing websites for clients on my own server. At that stage I definitely don’t want them indexed before the final version is uploaded to the client’s domain.

This is also useful for any part of your own site that you wish to see online, but not indexed until ready. Alternatively you might have folders with contents that you do not want appearing in search results. If you don’t want something exposed to the world through search engines, tell the robots not to crawl it.

1 Like