IndexNow protocol for crawling websites

Hi all,

Just noticed this and thought it might be worth a read for you guys, could come in useful for getting new content indexed more quickly.

1 Like

Thanks that is interesting to note and suggests they are aware of the problem. For about 6 months now it’s been taking ages to have new websites indexed and ranking even semi decently. It looks like we should be adding the API key to sites already.

Looks like Bing is working on their market share with changing indexing. With a marketshare growth of only 3% over 11 years, this might help to give it a boost.

EDIT: Open question. @Norm, should/could this be baked into Blocs similar to google? :thinking:

I’ve found the same thing @Flashman even with high quality links, things are really slow. Even sites which are well established, the index is slow to respond to modified and improved content, making it very difficult to sell seo etc.

I’m not sure if Google will support it officially, but it’s a fair assumption they’ll take any advantage to tweak their service and reduce workload their end. I’m sure there will be a way for them to use it to filter out spam sites etc.

I got a heads up from Cloudflare today that indexnow is automatically included with all their plans. As an aside, I also believe the service they offer for converting jpg to webp, will soon include avif.

1 Like

Interesting. Even more so that a ping method hasn’t been used before.

@DaveC From recent experience new sites have not even been indexed after a few weeks despite some good backlinks unless they were added to the Google search console. Adding them there seems to speed up initial indexing, but thereafter placing anywhere useful in the rankings is taking forever and as you say it makes any claim to providing seo services almost implausible.

It looks like IndexNow will be important and I think we need some means of making this easy in Blocs.

1 Like

Yes I think this would be a killer feature for Blocs to include out the box.

Probably a necessity given that all the major platforms are set up to include it as standard. It seems this has been planned for quite some time and I wondered if this might even be related to the slow ranking updates this year.

I’ve been going through the link and I don’t think Norm will be able to implement this for plain HTML sites. The indexing call towards the IndexNow protocol needs to be dynamic. So you’d either need to have a CMS doing the call or be using a cron-job with a server-side script (perhaps a php-file) to run the call towards for example Bing or Yandex.

It uses this structure:
https://www.bing.com/IndexNow?url=url-changed&key=your-key

However, now I come to think about it, @Norm could probably build a function in Blocs that lists the urls based on the page-names and the website address set in the project settings. I just read in the documentation that the requests via the IndexNow protocol don’t need to be executed on the server.

3 Likes

I’ve just been looking at this and it’s going to be a pain if you have a lot of sites to set up without some kind of automated solution, especially if it is for client sites.

I am still studying this but it appears the site key first need to be verified through a search console like Bing or Google. On a side note I see Bing has something called Microsoft Clarity, which is a free heat map option to see how visitors are engaging with your website.

2 Likes

Yes I was thinking the same, I feel for you guys who work on sites for clients; fortunately I only have to deal with my own.

At the moment I’ve only activated this through Cloudflare to see if there’s any difference from November onwards.

The biggest ranking improvement I’ve seen recently, has been through reworking my site designs to fully embrace the new core web vitals requirements from Google.

Most have doubled in traffic and search impressions, but one has increased 4x simply by tweaking the layout a little.

These were all existing sites with A grade page speed tests, yet still failing the CRUX tests google now uses.

I have a little update that should be very good news for some of you. The developer of Scrutiny has added a feature that makes updates for IndexNow a piece of cake and I have already tried this successfully.

The instructions are here but essentially you obtain a single developer API code inside Bing webmaster tools and paste it to the preferences of Scrutiny, which covers all your websites registered with Bing. Now run a scan and choose the option to export Bing Webmaster XML.

That will ping the update over to Bing and you can go back to drinking your pina colada on the beach and put the laptop under the sun lounger. I believe it is due some further refinements that should make it even better.

3 Likes

Awesome news @Flashman and thanks for reminding me! :desert_island: :sunglasses: :sunny:

My daily routine!