Why Use Prerender Services?
Prerender services crawl a website as a normal visitor. Using their own user-agent. They download the DOM (document object model) and then cache the HTML on their servers. When a search engine bot requests the page from the web host, the request is intercepted by the prerender service and a server-side cached version is served instead of the client-side rendered version.
I won’t expose the website where I encountered a prerender service for the first time. It consisted of an “index.html” shell page which would load React and client-side render the page according to the URL and backend API calls. By default, Google wasn’t able to crawl the DOM HTML; prerendering was required to index the pages in the search engine.
Here are all the headaches I encountered when I used a prerender service.
There was a routing failure which caused search engine requests to not redirect to the prerender service. As a result, our client-side rendered pages were indexed improperly. Google was only able to crawl the page source of the “index.html” shell page instead of the DOM. Showing a generic title and a weird description snippet. Once the routing was restored, Google was able to crawl URLs correctly and traffic returned to normal levels.
CSS Parsing Problems
When I logged into the prerender service interface and previewed a number of our pages, I discovered that their appearance varied very differently than what the user was viewing. The layout looked like a giant jigsaw puzzle, and this was being served to Google. The problem was that the prerender service wasn’t able to parse certain CSS selectors on the page. They solved the problem by implementing a custom solution for our account.
Desktop vs. Mobile CSS Intricacies
Some React websites are programmed to dynamically render for mobile users instead of relying on CSS breakpoints. This could cause desktop pages to be served to Google smartphone bots, causing the search engine to conclude that pages aren’t mobile-friendly. I can’t say that I encountered this issue, but it’s possible for this problem to occur if a React site doesn’t depend on CSS @media breakpoints.
Modals Are Cached Instead of Page Content and Features
If a pop-up modal becomes the main focus of the window, then it’s possible for the prerender user-agent to only crawl the HTML of the pop-up. Preventing Google from being able to crawl content and links in the DOM, which could cause pages from ranking for search terms.
A/B Testing Experiences Get Indexed
Most A/B testing platforms exclude search engine bots from getting experiment versions. However, prerender user-agents aren’t excluded by default. Therefore, it’s possible for Google to detect experiments and treat them as part of the page’s HTML. If the variations aren’t SEO-friendly, then its rank can decrease.
Are There Prerender Alternatives?
These services were discovered by visiting Stackshare. A CDN could be a viable alternative if the server’s able to intercept the search bot request and serve an HTML server-side version for crawling.