IndexNow is a major change in search engine indexing



Microsoft announced IndexNow, a major evolution in the way all search engines can discover and index content. IndexNow is open source and open to all participating search engines. When content is published or updated on a website, IndexNow notifies all participating search engines, dramatically reducing indexing time.

IndexNow sharing between search engines will begin in November 2021. There is a limit of 10,000 URLs per day.

The IndexNow protocol is open to all search engines with a significant presence in at least one market.

Evolution of Search Indexing from Pull to Push

There are two ways for search engines to get web page data: pull and push.
Pull is when a search engine robot visits a site to request web pages and “pulls” data from the server. This is how search engines traditionally work.

What IndexNow does is change content discovery to a push method where a CMS like WordPress tells search engines when content has been published or updated.


Continue reading below

The benefit for publishers is that they no longer have to wait for search engines to crawl and discover content.

Search engines know this right away and visit the site to pick up new or updated pages.

According to Microsoft’s announcement:

“… historically, one of the biggest problems faced by website owners has been allowing search engines to quickly discover and review the latest changes to their website.

It can take days or even weeks for new URLs to be discovered and indexed in search engines, resulting in loss of potential traffic, customers, and even sales.

… Once search engines are notified of updates, they quickly crawl and reflect website changes in their index and search results.

Who is behind IndexNow

IndexNow was created by Microsoft and Yandex. The protocol is open source and open to all search engines to participate.

When a search engine is notified of an updated or changed web page, all participating search engines will be notified.


Continue reading below

“IndexNow is a new protocol created by Microsoft Bing and Yandex, allowing websites to easily notify search engines whenever their website content is created, updated or deleted. Using an API “

How IndexNow benefits publishers

IndexNow benefits publishers because it will reduce crawl requests on the server, search engines will not need to do exploratory crawls to check if web pages have been updated and this reduces the time it takes to update. discovery and indexing of content.

Reducing server load helps the server perform optimally without the added load of redundant web page service that search engines already have.

Ultimately, this benefits the world by reducing the energy needs of exploration and indexing and helping to reduce the pressures of global warming.

Bing Announcement Notes:

“IndexNow is a smarter Internet initiative… website owners provide a clear signal helping search engines prioritize crawling these URLs, thus limiting the need for exploratory crawling to test if content has changed…

In the future, search engines intend to limit crawling of websites adopting IndexNow. “

Large companies adopt the new protocol

Microsoft has announced that large companies like “eBay, LinkedIn, MSN, GitHub, and Bizapedia” are planning to migrate to IndexNow from the Bing URL Submission API.

It is assumed that other large companies will follow suit as the benefits of the new system are clear.

For this reason, small businesses may consider adopting the new protocol as well in order to remain competitive and improve their indexation.

How does the new protocol work?

There are three suggested ways to submit URLs.

1. SEO integration of IndexNow

The first way, OnCrawl and Botify intend to integrate IndexNow.

2. Direct submission via API

The second way is a direct submission via an API.

Developers can use an API to automatically submit URLs to IndexNow.

The ad describes it as follows:

“1. Generate a protocol supported key using our online key generation tool.
2. Host the key in a text file named with the key value at the root of your website.
3. Start submitting URLs when your URLs are added, updated, or deleted. You can submit a URL or a set of URLs per API call.

Submitting a URL is as easy as sending a simple HTTP request containing the modified URL and your key."


Continue reading below

3. Integration with CMS and CDN

But the third way may be how most sites will interface with IndexNow, accessing the new protocol through a native feature at the heart of the CMS that a publisher uses.

Wix and Duda recognize the benefits for users and are already planning to provide the new functionality to their users.

Cloudflare and Akamai also support the new IndexNow protocol.

Waiting for WordPress?

While competitors like Wix and Duda have recognized the importance of IndexNow, WordPress seems to be dragging its feet on the new protocol.

Maybe WordPress will adopt it, but over the past seven months, major developers have discussed the proposal with no apparent resolution.

According to the WordPress public development ticket, there does not appear to be a decision on this yet.

Nonetheless, Microsoft submitted the IndexNow open source code to WordPress for inclusion in the kernel.

If adopted by WordPress, this will make it automatic that a publisher can choose to submit their URLs through IndexNow.


Continue reading below

In response to a question from core WordPress developers, Microsoft revealed that publishers will be able to choose which search engines to submit their changes to and new URLs to.

This means that they can block specific search engines from receiving data about their pages.

According to the WordPress development ticket, the Microsoft representative said:

“WordPress and other CMS admins will be able to select which industry players they want to notify, including everyone through centralized services if they want to notify everyone. “

IndexNow is the evolution of search indexing

IndexNow is a big deal because it is a major change in the way search engines can discover updated and newly published documents.

It benefits publishers with fast indexing and reduced server load from bots constantly crawling their web pages.

Adoption by major search engines will be an evolution of the relationship between publishers and search engines that benefits both.


Read Microsoft’s announcement

IndexNow Listing – Instantly Index Your Web Content In Search Engines


Continue reading below

Visit the IndexNow website

WordPress IndexNow Development Ticket and Extract Request

Instantly index WordPress website content in search engines

Official IndexNow WordPress Extract Request


Leave A Reply

Your email address will not be published.