In a recent conversation with Google’s John Mueller clarified inquiries linked to indexing. He reveals how website quality plays a critical role in influencing indexing models. Later with the dialogue, he acknowledged that 20% of the website content is not indexed. 

Pages located yet not crawled.

The person who raised the inquiry presented information about their website. The overburdened server points to some serious consequences on the number of pages google indexes.

The person’s inquiry:

20% of my pages are not getting indexed.

It says they’re discovered but not crawled.

Does this have anything to do with the fact that it’s not crawled because of the potential overload of my server?

Or does it have to do with the quality of the page?”

So, what happens when a server is overburdened? The screen displays a 500 error response when a web page is inquired. Basically, a 500 response error occurs when the internet is incompetent to serve a web page. 

The person failed to point out the query clearly. The individual didn’t mention that Google Search Console communicating that Googlebot was getting 500 error response codes. 

And if Googlebot is not behind the 500 response error. Then the answer to why 20% of the web pages did not get indexed is not the overburdening of the server.

Crawl Funds is not the answer to why Small Sites Have Non-indexed Pages.

John Mueller’s reply to the query:

“Probably a little of both.

So usually if we’re talking about a smaller site then it’s mostly not a case that we’re limited by the crawling capacity, which is the crawl budget side of things.

If we’re talking about a site that has millions of pages, then that’s something where I would consider looking at the crawl budget side of things.

But smaller sites probably less so.”

After hearing the query, Google’s John Mueller came up with an impressive explanation that clears out how website high quality influences Googlebot decision of indexing internet pages. 

But, before explaining any further, he suggested that the crawl budget is not the answer to why small sites have non-indexed pages. 

The Google indexing decision is highly influenced by overall sites quality

One part of John’s explanation tells that how a web page crawled and indexed is affected by the high quality of a web page. The answer came out exciting because of the revelation made concerning how Google evaluates web page quality. 

Indexing and Technicalities

Mueller continued:

“Because I think, for the most part, sites nowadays are technically reasonable.

If you’re using a common CMS then it’s really hard to do something really wrong.

And it’s often more a matter of the overall quality.”

He further disclosed the technical features of indexing. He added that currently, it is easy for websites to get the correct part so that their indexing remains intact.

Its usual for a site to not be indexed

Mueller further revealed: 

“The other thing to keep in mind with regards to indexing, is it’s completely normal that we don’t index everything off of the website.

So if you look at any larger website or any even midsize or smaller website, you’ll see fluctuations in indexing.

It’ll go up and down and it’s never going to be the case that we index 100% of everything that’s on a website.

So if you have a hundred pages and (I don’t know) 80 of them are being indexed, then I wouldn’t see that as being a problem that you need to fix.

That’s sometimes just how it is for the moment.

And over time, when you get to like 200 pages on your website and we index 180 of them, then that percentage gets a little bit smaller.

But it’s always going to be the case that we don’t index 100% of everything that we know about.”

The concluding part of John Mueller’s explanation is remarkable when he says that 20% of a website is not indexed being in normal bounds. 

It is absolutely okay if your page is not indexed

John Mueller’s explanation includes several pointers about indexing that people need to grasp.

1. Technical aspects of a website have no effect on indexing.
2. The quality of a site can influence indexing decisions.
3. The percentage of a site indexing keeps on fluctuating.
4. Small sites have not to fear crawl budgets