There are a lot of websites created everyday. And at any given moment, large websites increase their number of pages, sometimes significantly.
For search engines, crawlers have a lot of job to do. And indexing the World Wide Web isn't an easy task, considering the huge amount of pages that are added every single day.
For this reason, Microsoft that operates the search engine Bing, partners with Botify, an enterprise SEO platform.
With the partnership, Bing can bring near real-time indexing (faster than XML submition) to websites through the Bing content submission API pilot.
Called the 'FastIndex', which is part of Botify Activation, it provides Bing new content feed, and increased its API limit to more than 10,000 URLs.
This should be beneficial for large websites with a lot of content updates, and those that have 10,000 pages or more, and those that publish new content on a regular basis.
Those sites can include, and not limited to: large news websites, retail and e-commerce sites, classifieds sites and marketplaces, among other large content publishers.
According to Botify in its announcement blog post:
"We’re fixing that."
Through the launch of FastIndex, which is built upon Bing’s updated programmatic URL submission process, websites can submit their newest, most critical pages directly to Bing.
With the ability for Bing to do immediate indexing, large website owners should benefit the most.
What's more, this should give them more control over content and giving them the ability to create better campaign for better coverage.
According to Adrien Menard co-founder and CEO of Botify, Bing already benefits from lower costs vs. crawling and presumably, higher quality content. However. Menard observed that it can take Bing a couple of days before it can properly index new content. He also said that Bing may for time to time fail to index "more than half of a typical enterprise website’s pages."
“Search engines don’t have resources to crawl every page of the internet, and websites are changing all the time,” said Menard. But with the API, “You can select the pages to be indexed.”
And on top of the above, Botify can also analyze websites and their content, and make recommendations about which of their pages should or should not be submitted.
The web is growing, and it will continue to grow as more websites and more servers are put up online.
As a search engine competing with Google, Bing has a slight advantage. For example, Bing API can accept core site content, if compared to Google's indexing API which only has a more limited number of content types.
And this Bing and Botify partnership is giving Bing yet another advantage. But depending on how successful this partnership will turn out, Google may decide to follow Bing's footstep, and open a wider range of content to its API.
Botify’s customers already include Macy’s, Expedia, Condé Nast, eBay, Monster and others. In the meantime, Bing is testing the API with a few large partners to see how it goes.