What crawler do you recommend for finding orphaned pages on a website?
-
Is there a crawler that you guys recommend for finding all pages, including orphaned pages on a website? A data export is not feasible. I saw a question from back in 2013 and was wondering if anything has changed since then in regards to crawling orphaned pages. Do most enterprise systems already have this built into their crawler? Or is it best to get a crawler like Xenu or Screaming Frog or Deepcrawl?
-
Hi there!
i agree with Patrick. I was going to recommend using Screaming Frog or Google Search Console! Let me know if you try these, don't like them, and need another recommendation.
-
Hi there
I really like ScreamingFrog but I also really like Search Console and Moz Pro. The reason being, I like having different sets of data because they are all different. I also like seeing if pages are being linked to randomly from other sources other than my own website which Search Console does a great job (and so does Majestic or Ahrefs). Different sources find different things so it's nice to get other opinions on what you might have out there floating around.
Just my two cents! Hope this helps!
Patrick
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Authority for localized version of website
Hello everyone, I have a case here were I need to decide which steps to take to improve page authority (and thus SEO value) for the German pages on our site. We localized the English version into German at the beginning of 2015. www.memoq.com - English de.memoq.com - German By October 2015 we implemented href tags so that Google would index the pages according to their language. That implementation has been successful. There is one issue though: At that time, all our localized pages had only "1" point for Page Authority ("PA" in the Moz bar). At the beginning we though that this could be due to the fact that localization was done using subdomains (de.memoq.com) rather that subfolders (www.memoq.com/de). However, we decided not to implement changes and to let Google assess the work we had done with the href tags. Its been a while now, and still all our German pages keep having only "1" point for page authority. Plus we have keywords for which we rank in the top 10 in English (US Google Search), but this not the case for the translated version of the keywords for German (Germany Google search). So my question basically is: Is this lack of page authority and SEO value rooted in the fact that we used subdomain instead of subfolder for the URL creation. If so is it likely that Page Authority for German pages and SEO value will increase if I change the structure from subdomains to subfolders? Or is it that the problem in PA is rooted somewhere else that I am missing? I appreciate your feedback.
Technical SEO | | Kilgray0 -
How do I find which pages are being deindexed on a large site?
Is there an easy way or any way to get a list of all deindexed pages? Thanks for reading!
Technical SEO | | DA20130 -
Pageing page and seo meta tag questions
Hi if i am using paging in my website there is lots of product in my website now in paging total paging is 1000 pages now what title tag i need to add for every paging page or is there any good way we can tell search engine all page or same ?
Technical SEO | | constructionhelpline0 -
Old Product Pages
Hi Issue: I have old versions of a product page in the Google index for a product that I still carry. Why: The URLs were changed when we updated this product page a few years ago. There are four different URLs for this product -- no duplicate content issues b/c we updated the product info, Title tags, etc. So I have a few pages indexed by Google for a particular product. Including a current, up-to-date page. The old pages don't get any traffic, but if I type in google search: "product name" site:store.com then all of the versions of this page appear. The old pages don't have any links to them, only one has any PA, and as I said they don't get any traffic, and the current page is around #8 in google for its keyword. Question: Do these old pages need 301 redirects, should I ask google to remove the old URLs? It seems like Google picks the right version of this page for this keyword query, is it possible that the existence of these other pages (that are not nearly as optimized for the keyword) drag it down a bit in the results? Thanks in advance for any help
Technical SEO | | IOSC0 -
SEOMoz Crawl Diagnostic indicates duplicate page content for home page?
My first SEOMoz Crawl Diagnostic report for my website indicates duplicate page content for my home page. It lists the home page URL Page Title and URL twice. How do I go about diagnosing this? Is the problem related to the following code that is in my .htaccess file? (The purpose of the code was to redirect any non "www" backlink referrals to the "www" version of the domain.) RewriteCond %{HTTP_HOST} ^whatever.com [NC]
Technical SEO | | Linesides
RewriteRule ^(.*)$ http://www.whatever.com/$1 [L,R=301] Should I get rid of the "http" reference in the second line? Related to this is a notice in the "Crawl Notices Found" -- "301 Permanent redirect" which shows my home page title as "http://whatever.com" and shows the redirect address as http://http://www.whatever.com/ I'm guessing this problem is again related to the redirect code I'm using. Also... The report indicates duplicate content for those links that have different parameters added to the URL i.e. http://www.whatever.com?marker=Blah Blah&markerzoom=13 If I set up a canonical reference for the page, will this fix this? Thank you.0 -
Page crawling is only seeing a portion of the pages. Any Advice?
last couple of page crawls have returned 14 out of 35 pages. Is there any suggestions I can take.
Technical SEO | | cubetech0 -
Looking to hire someone to help on my website. On page issues with subpages outranking homepage
Hello, I have a site that has been doing good for quite a while now. But lately I'm running into issues with on page seo. I have ranked well for long periods of time, but changes like adding content or moving stuff around has bumped me out of the rankings. I would prefer not to publicly disclose the site. My main keyword I rank for is "word1 word2". I notice if I do a site: search for "word1 word2" homepage is #1, site search for just word1 it's #1 as well, but the word2 returns a category listing above the homepage. I noticed this happen right as I fell back from the rankings. I am looking to hire an SEO profession to help me out with on page issues. I've come to admit that I just don't understand something about this and I need help from someone who has a lot of experience with this. I was hoping to find recommendations by people here for a company I could hire for this. I would want to pay no more than $1,000. I hope that budget is high enough to get a skilled individual/company working with me. I'm not sure where to search for online to get a skilled on page seo professional, so I thought a personal recommendation from someone here would possibly be able to help. Thanks
Technical SEO | | nux0 -
On-Page Question
Im trying to increase value to specific pages by putting history, and additional images. Will copying snippets from other sites negatively affect me? Should the content be re-written completely?
Technical SEO | | Anest0
