One of the key lessons I’ve learned is never to rely solely on Screaming Frog or any other crawling tool to discover all the pages of a website (which I used to think was the case too! 😄).
Screaming Frog and similar tools rely on internal links to discover new pages.
If you have thousands of pages with poor internal linking or outdated sitemaps, Screaming Frog won’t be able to detect them.
For example, when Screaming Frog was connected to Google Search Console (GSC), GA4, and provided with sitemaps, it detected around 1,600 pages.
Without connecting to GSC and GA4, it only discovered about 1,300 pages.
I remember working on a large enterprise website with no functional sitemaps and poor internal linking.
Screaming Frog detected around 2,500 pages without GSC connected, but once GSC and GA4 were integrated, it discovered around 6,000 pages. However, when I accessed the website’s dashboard, the actual number of pages was significantly higher than what Screaming Frog found.
To get the most comprehensive crawl, always connect Screaming Frog to Google Search Console, GA4, and ensure sitemaps are provided.