Emulating Googlebot Behavior in Chrome is not a thing(sorry to say that)
What I Did:
-I used Chrome’s DevTools and extensions to emulate Googlebot to see how Google views the page.
What Happened:
On the category page, it showed that the JavaScript redirect was working correctly. This led me to believe that Google could follow the redirect seamlessly.
The Problem:
When I checked Google Search Console (GSC), the page was empty with no products. Initially, I thought Google had a rendering issue, but after testing, I discovered the following:
-For regular users: The website uses JavaScript to redirect to a parameterized URL.
-For Googlebot: The website serves the original page without any redirect.
- Different Versions for Googlebot and Users
The Misunderstanding:
The Chrome emulation made me think that both users and Googlebot were seeing the same page, but this wasn’t true.
What I Discovered:
Googlebot does not follow the redirect as the normal user does. So, while normal users were redirected to the correct product page, Googlebot was shown an empty page.
Lesson Learned:
Chrome’s Googlebot emulation can miss important differences between what real users and Googlebot see, leading to misleading conclusions.
- HTTP Status Codes Can Be Deceptive
What Happened:
When I inspected the page in GSC, I noticed that the HTTP status code was 200 (OK), which made it seem like everything was fine. But in reality, Google was being served an empty page.
Why I Missed It:
I was relying on the emulation tools, which followed the redirect, making it look like everything was working perfectly.
Key Takeaway:
Always cross-check with Google Search Console—don’t rely solely on Chrome’s emulation tools.
- Using Screaming Frog Revealed JavaScript Issues
What I Did:
I used Screaming Frog to further investigate why Googlebot was seeing an empty page.
Initial Results:
Screaming Frog also showed an empty page, but when I set it to ignore robots.txt, it rendered the products correctly.
Discovery:
A JavaScript file responsible for serving the products was blocked by robots.txt.
Lesson Learned:
Robots.txt files can block essential scripts, which can prevent products from being displayed correctly in crawlers like Screaming Frog and Googlebot.
- Subdomains Have Separate Robots.txt Rules
The Issue:
I tried using a custom robots.txt to allow certain scripts, but Screaming Frog kept showing that they were blocked.
What I Discovered:
The scripts were being served from subdomains with their own robots.txt files, and even though I allowed them in the main domain’s robots.txt, Google respects subdomains’ robots.txt rules independently.
Key Takeaway:
Google respects robots.txt rules at the subdomain level. Even if you allow scripts on the main domain, they can still be blocked by the robots.txt on subdomains.
- google Emulating in chrome is not a thing to consider