Member Login
menu

Websites With 'Infinite Scrolling' Pages, And How Google Crawls The Unlimited

When websites need to deal with massive contents or posts that have to be displayed inside one single web page, there are two common ways they can follow.

First, they can use what's called the pagination feature. This feature is classic and straightforward.

When the number of posts on a web page reaches a certain limit that is defined by the developer, the contents will be broken down to multiple pages. Users could navigate through the posts by going through the pages by clicking or tapping on the pagination number.

Second, they can use 'infinite scrolling', where contents and posts will load as soon as users reach the end of the page.

Infinite scrolling is also known by other names, like "endless scrolling," or "continuous scrolling."

This feature is popularized by social media networks, where it allows users to scroll and scroll, and continue to scroll in order for the platform to increase engagements.

Continuous scrolling

While this feature is certainly appealing and sometimes mesmerizing as contents load without end, it does pose a bit of a problem for Google.

Google has what it calls the Googlebot, a crawler that the company uses to crawl web pages.

While infinite scrolling means that the page can have "unlimited" contents or posts, Google does crawl those pages.

But only to a certain extent.

Google requires resources to jump from one website to the next, crawling their pages whenever it see them.

But when dealing with pages with contents that keep on loading as Google tries to index, Google will have stop after a while, and call it a quit.

This topic was discussed during the Google Search Central SEO office hours hangout recorded on February 18, 2022.

Someone asked Google’s Search Advocate John Mueller a series of questions about paginated content, and one of the questions involves infinite scrolling, and whether Googlebot has advanced to the point of being able to handle it.

Read: How Search Engines Process Your Queries Determines Your Satisfaction

"A little bit… I thought we had a page on that but I think we might not actually have finished that up."

Mueller added:

"Essentially what happens when we render a page is we use a fairly high viewport, like if you have a really long screen. And we render the page to see what the page would show.

And usually that would trigger some amount of infinite scrolling in whatever JavaScript methods that you’re using to trigger the infinite scrolling. And whatever ends up being loaded there that would be what we would be able to index.

So that’s something where, depending on how you implement infinite scroll, it can happen that we have kind of this longer page in the index."

This explains how Googlebot is able to crawl web pages with infinite scrolling, but cap to the amount of viewport Google has assigned the crawler to follow.

In order words, Googlebot may not be able to render and crawl everything an infinite scroll page has.

"It might not be that we have everything that would fit into that page because, depending on how you trigger infinite scroll, it might be that oh you’re just loading the next page kind of thing.

And then we might have two or three of these pages loaded on one page with infinite scroll, but not everything.

The best way to know how far Googlebot can fetch the content of a page, is by using the URL Inspection tool, Mueller explained.