Recently, in a Google-SEO office-hours hangout, John Mueller, a senior webmaster trends analyst at Google, was asked about the 500 response and the role of robots.txt in indexing a web page.
In the session, the publisher made a point about a bug hindering the validation process of rich results. The bug fails to notify the publisher about the error and passes all the information with a successful validation.
Google Search Console and Rich Results
Google Search Console helps web admins check the indexing status and optimize their site’s visibility. In January 2018, Google launched a new version of this search console which has met a few criticisms in recent years.
On the other hand, the rich result is a new name for the Google unique search that includes snippets, carousels, and other elements. They provide an overall better experience on Google Surfaces.
See Also: Google’s Untitled Search Results Sending Users To Spam Sites, Google ‘Working On It’
The case of missing images in the previews
In the hangout video, the publisher explained the whole issue. After checking all the metadata, he said that Google Search Console passes the rich recipe content by saying it’s all good.
However, the webmasters still found missing images in the preview results. He further highlighted the issue of 500 response errors when CDN requests robots.txt.
In answer to the above query, Mr. Mueller agreed. He remarked that “from our point of view, it’s not that a robots.txt file is required. But it also needs to have a proper result code.”
“He further stated how Google’s systems assume some error regarding the server if a server error is returned for the robots.txt file.
In answer to the rich results test, Mr. Mueller said that as far as he knows, the rich results test focuses on content that’s on HTML pages. The JSON-LD markup, which is a process of encoding linked data for humans to read and write, fails to check whether the images are fetchable or not.
It led to no utilization of images in the Carousel too. He accepted the need for something to figure out how they should highlight it better.
500 Response on Robots.txt Fetch
When you get 500 responses on your page, the server encounters an unexpected internal error. The server can’t fulfill the client’s request when a 500 reply happens.
The primary reason for this is the absence of necessary things inside the code. Due to this, the server directly returns a 500 response when called.
It’s one of the rare problems that SEO members face, and there isn’t any permanent solution as of now. However, a short-term solution that works is to upload a robots.txt file.
Otherwise, you will have to check your server or CMS to figure out the underlying problems, If you face this problem in the future, you will now be able to know what is happening behind the scenes.