Hi, I have tried to read and understand what “robots.txt” is and why it’s even mentioned as “very very important.” The more I read and the more videos I check, the less i understand Can anyone explain (for a simple analog human) short why this file exists and why it’s needed or not needed. Right now I see I have a robots.txt on my website, in the root directory, but I have no idea how it got there or why it’s there. The text below is included in this text-file. Should robots.txt be there? My website has very very very few visitors even if the products offered there are all supa top class.
In practical terms if you don’t include a robots.txt file, search engines will just ignore or index what they feel like, based on what they find. On the other hand, you can use a robots file to either request the whole site is not indexed or maybe specify certain sections that should not be indexed.
You can even single out particular search engines or robots for particular treatment, though you are still reliant on their good graces to obey your wishes.
In the case of the robots file you show that translates like this:
That means any user agent. The * is a wildcard applying to all.
If there is anything you want disallowed i.e not indexed it should follow there and in this case you have nothing, so that combination of entries is basically telling all robots to index anything and everything.
Finally you just have a helpful pointer indicating where the sitemap is located.
Overall, it’s just good practice to include a robots.txt file. You also have some control via Blocs in the page settings.
Thanks, Flashman, this is tricky, I don’t really get the logic/functions in this. Let’s say like this: I want customers to easy find any product page at the website www.himmelstrutz.com. How does robots.txt help with this (what exactly should be written in it)?
Customers finding your pages and products easily is more about the site structure and creating the right user experience. This will also encourage Google to take your site more seriously.
The robots.txt file that you showed simply gives a green light for search robots to index anything they like and points them to your sitemap with a list of pages and images. Just remove “mywebsitename” from the sitemap and put your own site url in there.
Thanks again Flashman, I have/had my real website name in there, but after a few days using this robots.txt my website more or less disappeared when googling it, and then i removed the robots.txt and a few days later it was back when googling. So in my case it seems like it’s better without robots.txt?
It shouldn’t hurt your ranking and that may have been coincidental or even temporary, but if you are happy with the results without the robots.txt file I would stick with that.
I know it shouldn’t be a problem, but I always prefer to put the sitemap link above the robots instruction like this:
I figure there is always a risk a search engine will exclude the sitemap if I do it the other way around, though I know it shouldn’t be a problem.
Real results are always more important than theoretical best practices. I sometimes have clients come back to me regarding old websites I built years ago, asking if they can improve things for Google. When I then check and see they are already in first place for their area I have to advise them that any changes risk that position.
Thanks again Flashman. It’s difficult to handle this as I don’t understand why/what I’m doing with the robots.txt. It’s like a language I have never heard before. I also wonder what the things in italic below means (YOAST) in my robots.txt
Can anyone explain (for a simple analog human) short why this file exists and why it’s needed or not needed. Right now I see I have a robots.txt on my website, in the root directory, but I have no idea how it got there or why it’s there. The text below is included in this text-file. Should robots.txt be there? My website has very very very few visitors even if the products offered there are all supa top class.