How to find orphan pages

[KJH-HAC]

Hi all,

I've been checking these forums for an answer on how to find orphaned pages on my site and I can see a lot of people are saying that I should cross check the my XML sitemap against a Screaming Frog crawl of my site.

However, the sitemap is created using Screaming Frog in the first place... (I'm sure this is the case for a lot of people too).

Are there any other ways to get a full list of orphaned pages? I assume it would be a developer request but where can I ask them to look / extract?

Thanks!

[Roman-Delcarmen]

Yes I mentioned in my case I use Semrush and there is a dedicated space for that specific parameter. The easiest way to get your log files is logging into your cPanel and find an option called Raw Log Files. If you are still not able to find it, you may need to contact your hosting provider and ask them to provide the log files for your site.

Raw Access Logs allow you to see what the visits to your website were without displaying graphs, charts, or other graphics. You can use the Raw Access Logs menu to download a zipped version of the server’s access log for your site. This can be very useful when you want to quickly see who has visited your site.

Raw logs may only contain a few hours’ worths of data because they are discarded after the system processes them. However, if archiving is enabled, the system archives the raw log data before the system discards it. So go ahead and ensure that you are archiving!

Once you have your log file ready to go, you now need to gather the other data set of pages that can be crawled by Google, using Screaming Frog.

Crawl Your Pages with Screaming Frog SEO Spider

Using the Screaming Frog SEO Spider, you can crawl your website as Googlebot would, and export a list of all the URLs that were found.

Once you have Screaming Frog ready, first ensure that your crawl Mode is set to the default ‘Spider’.

Then make sure that under Configuration > Spider, ‘Check External Links’ is unchecked, to avoid unnecessary external site crawling.

Now you can type in your website URL, and click Start.

Once the crawl is complete, simply
a. Navigate to the Internal tab.
b. Filter by HTML.
c. Click Export.
d. Save in .csv format.

Now you should have two sets of URL data, both in .csv format:
All you need to do now is compare the URL data from the two .csv files, and find the URLs that were not crawlable.

If you decided to analyze a log file instead, you can use the Screaming Frog SEO Log File Analyser to uncover our orphan pages. (Keep in mind that Log File Analyzer is not the same tool that SEO spyder)

The tool is very easy to use (download here), from the dashboard you have the ability to import the two data sets that you need to analyze

If the answer were useful do not forget to mark it as a good answer ....Good Luck

[KJH-HAC]

Hi Roman,

Out of interest, is there an option to expert an orphan page report like there is in Screaming Frog? (Reports / Orphan Pages).

I guess the true and most realistic option is to get the list from the dev team as using the sitemap isn't plausible as these pages should still get indexed. The new Google Search Console also lets you test individual pages and as long as they're in the sitemap, they should (hopefully) be indexed.

Still, trying to get a list of ALL pages on a site, without dev support, seems to be a challenge I'm trying to solve

[Roman-Delcarmen]

Even Screaming-frog have problems to find all the orphan-pages, I use Screaming-frog, Moz, Semrush, Ahrefs, and Raven-tools in my day to day and honestly, Semrush is the one that gives me better results for that specific tasks. As an experience, I can say that a few months ago I took a website and it was a complete disaster, no sitemap, no canonical tags, no meta-tags and etc.

I run screaming-frog and showed me just 200 pages but I knew it was too much more at the end I founded 5k pages with Semrush, probably even the crawler of screaming frog has problems with that website so I commenting that as an experience.