If I use an app like Scrutiny to create an xml sitemap it first goes to the home page and crawls through other pages, however any page without a direct internal link is missed and this happens with Google as well. In other words standalone pages without an incoming link from somewhere on the site are lost. This is common if they are not in the global menu or footer.
I know Blocs will export a XML sitemap, but we almost need a feature that will create a webpage with a sitemap or index of contents that can be picked up to ensure that every page is indexed.
This is something I seem to recall was in Freeway at least 10 years ago and I think it was in Dreamweaver of GoLive before that. Yes we could do this manually, but on a large site it is potentially very time consuming and easy to forget a page or two.
All it would require is a feature that exports a list of pages as links to a sitemap page.
Yes that would do the trick, but this is different from the sitemap that gets uploaded to the server as a xml document. This is basically a web page with with the listed pages and each entry is a link to the corresponding pages.
You often see these on more old fashioned sites and less often these days, but I can’t think of a better alternative if you want to ensure that pages have an incoming link from another part of the site that will help them get indexed.
Gotcha, yeah used to be the thing to have a page of links of “friends” back in the day too.
Unless Norm builds that in, probably an option is for someone to make a Bric that reads the default sitemap XML file and populates a page nice and fancy.
The incoming links from friends wouldn’t work here unless they went direct to those pages, so by far the best option is some kind of link list on the website itself.
The reason I’ve thought of this is that I want to rebuild my site and there will be a lot of standalone pages not listed in the main menu or footer, so I need a means of getting those pages indexed by Google.
At the moment I’ll just find a secluded spot like the FAQ and include a link to this internal sitemap page, so it all links up nicely. It can be built by hand, but some sort of automated process would save a lot of time and reduce the risk of error.
It can be added in all sorts of ways. I didn’t want to add a sitemap link to the footer and it just needs some discreet means of ensuring those pages are picked up.
I know this topic is over two years old, but the original request is still valid. When you are working on a website that is well over 100 pages not everything is going to be in the main menus and there will be pages with no direct link from other pages for various reasons, so they are potentially missed by a xml sitemap scan and Google.
This could all be fixed by Blocs generating an internal sitemap page that is automatically updated as pages change, including links to every page. This is currently a painfully slow manual task in Blocs.
From memory, Freeway also produced a visual link map that quickly highlighted any broken links within a project. I tend to pick up errors like this later on using Scrutiny, however could do with some kind of internal means to flag incorrect or broken links.
I don’t use the Blocs sitemap. It is much improved now with version 4, however it lacks some of the deep scanning abilities found in Scrutiny, which is dedicated to the single task of checking websites with forensic detail and exporting sitemaps.
Scrutiny helped me recently to uncover a bug in Blocs, which was leaving 404 not found errors in the CSS for deleted background images. That led to a fix in Blocs that now carries out checks for images in custom classes after an image is removed from the asset manager.
I tried exporting a sitemap just now with Blocs and saw it did not include an image added as a remote asset, so in that sense it is not giving a full picture. Scrutiny also includes media types such as video and PDF files etc that have been hosted remotely and uploaded separately.
It seems to me that if you also include a page on the site with a full list of internal page links that should at least help to ensure Google doesn’t hit any dead ends, while improving the odds of those pages being indexed. Google may otherwise see pages in a xml sitemap, but decide not to index them because no other internal page leads there.
The main reason I want this is because I’ll be setting up a number of landing pages for specific purposes. They are not something most visitors would see if they arrive on the home page, but they can still useful for driving traffic through search queries or targeted marketing.
Oh yeah, before I forget: You’re able to add more than one sitemap in Google search console. I am building a dynamic one for Volt blog posts like this one:
It doesn’t but I used it as a point to illustrate how detailed the checks are in Scrutiny. That bug would otherwise have remained hidden to all, except Google.
It seems logical that Google will be more likely to index pages that can reached from other destinations within the site as a basic point of usability. These landing pages all include links to other pages on the site, however they have no internal incoming links, unless I set up an internal page sitemap like this.
Yes I had read about those. From what I can see this is all handled in a single sitemap by Scrutiny, though I may just ask the developer, since she would know for sure. The one point I have been able to establish is that highly detailed sitemaps definitely lead to more of the site being indexed and therefore more chances to be picked up in searches. My rankings have generally improved when sitemaps include more information.
@Jannis I now recall that Google wants xml files below a certain size with a max number of urls, though I can’t remember the numbers off the top of my head. In that situation with a large website you must generate multiple xml sitemaps, then tie them together with a sitemap index file.
Scrutiny handles this automatically; generating the multiple files if one would be too large. If the sitemap size is below that limit everything is included in a single file. It’s very good to see Volt generating these sitemaps as well.