Adding robots.txt

I Need Help
1

So, found a couple of old topics thread about robots.txt (but I assume I shouldn’t revive/”necro” post) … Unless I’m missing something vital, it should be good to add a robots.txt to my published project, something like:

Sitemap: https://mywebsitename/sitemap.xml

User-agent: *
Disallow:

… As there’s nothing in particular to hide. (No VoltCMS or anything on this particular project.)

Should I just create the file in an editor and add to Project as Project Attachment under Project Settings? :thinking: :nerd_face:

2

Sitemap: https://mywebsitename/sitemap.xml

table of contents for your website. It lists all (or most) of your pages so search engines can easily find them.

User-agent: *

The * means all search engine bots.

Examples include:

  • Googlebot
  • Bingbot
  • DuckDuckBot
  • Yahoo’s crawler

Disallow:

This means Nothing is blocked. Search engines are allowed to crawl every page.

Overall benefit

For SEO, this setup is:

  • :white_check_mark: Search engines know where your pages are.
  • :white_check_mark: Nothing important is accidentally hidden.
  • :white_check_mark: New content is easier to discover.
  • :white_check_mark: Follows standard SEO practices.

Rated 9/10 for a typical public website.

1 Like
3

I don’t believe Blocs supports adding text files unless if under a folder. I could see no way of doing so and there was zero response from @Norm or anyone else to my query a couple of years ago.

:crossed_fingers:it’s in a development backlog somewhere.

Yes, you should simply create the file and put it on your website manually. If you do anything to sync your website from local files, you’ll want to ensure that it doesn’t get removed during the sync process.

I use Forklift for my website syncing and up until recently I would manually copy over my robots.txt to my website. These days I now do a both ways sync to ensure that my robots.txt on my website makes it locally to my git repo rather than getting removed during a one way sync.

2 Likes
4

Yes, I use Forklift, too. I guess I can set immutable flag on the robots.txt file to prevent it from being deleted. :sweat_smile:

2 Likes