What crawler do you recommend for finding orphaned pages on a website?
-
Is there a crawler that you guys recommend for finding all pages, including orphaned pages on a website? A data export is not feasible. I saw a question from back in 2013 and was wondering if anything has changed since then in regards to crawling orphaned pages. Do most enterprise systems already have this built into their crawler? Or is it best to get a crawler like Xenu or Screaming Frog or Deepcrawl?
-
Hi there!
i agree with Patrick. I was going to recommend using Screaming Frog or Google Search Console! Let me know if you try these, don't like them, and need another recommendation.
-
Hi there
I really like ScreamingFrog but I also really like Search Console and Moz Pro. The reason being, I like having different sets of data because they are all different. I also like seeing if pages are being linked to randomly from other sources other than my own website which Search Console does a great job (and so does Majestic or Ahrefs). Different sources find different things so it's nice to get other opinions on what you might have out there floating around.
Just my two cents! Hope this helps!
Patrick
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Website's Home Page is Missing on Google SERP
Hi All, I have a WordPress website which has about 10-12 pages in total. When I search for the brand name on Google Search, the home page URL isn't appearing on the result pages while the rest of the pages are appearing. There're no issues with the canonicalization or meta titles/descriptions as such. What could possibly the reason behind this aberration? Looking forward to your advice! Cheers
Technical SEO | | ugorayan0 -
Is it easier to rank high with a front page than a landing page?
My product is laptop and of cause, I like to rank high for the keyword "laptop". Do any of you know if the search engines tends to rank a front page higher than a landing page? Eg. www.brand.com vs. www.brand.com/laptop
Technical SEO | | Debitoor0 -
Redirects for new website
Hi Moz community,
Technical SEO | | JSimmons17
I'm a fairly new SEO Specialist with a brand new website. We initially had a very basic holding website until the fully functional website was completed. I have to do some redirects as we have both .html and .php files & we don't want to lose SEO value for specific pages (like the index, news, etc). I also want to redirect from a www url to a non-www url. I am trying to accomplish redirects with the following code: RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.mywebsite.com [NC]
RewriteRule (.*) http://mywebsite.com/$1 [R=301,L] RedirectMatch 301 /index.html (.*).(php|html) http://mywebsite.com/index.php RedirectMatch 301 /cupcakes-slideshow/glutenfree-slideshow.html (.*).(php|html) http://mywebsite.com/gluten-and-glutenfree.php RedirectMatch 301 /press.html (.*).(php|html) http://mywebsite.com/news-and-reviews.php Please let me know if I am on the right track. Thanks so much in advance!0 -
Can I put the tag in the MasterPage of my ASP.NET website or does this need to be specific to each page?
Hi Moz Community, I am a designer/junior SEO'er and have been working with our web developer to setup SEO oriented redirects and the rel canonical tag on our ASP.NET page running MasterPages - www.tisbest.org. I know setting up an incorrect canonical tag can be devastating so I'm hoping for some guidance. Can we put the <title> </span>Charity Gift Cards | Donation Gift Ideas | TisBest Philanthropy</p> <p style="color: #5e5e5e; font-family: Helvetica, Arial, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; line-height: normal;"><span style="color: #5e5e5e;"> </span></p> <p style="color: #5e5e5e; font-family: Helvetica, Arial, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; line-height: normal;"><span style="color: #5e5e5e;"></title> Thanks! Chad
Technical SEO | | TisBest0 -
Duplicate Page Title & Content Penalty On Website Tonight Platform
I built my primary website on Website Tonight (WT) five years ago when I was a net newbie and I'm presently new to seomoz. The initial crawl indicated a problem with duplicate page title and duplicate content with my website home page in WT. It turns out that the WT platform makes you assign a file name to your homepage i.e: www.business.com/homepage.html that differs from the www.business.com that you want as your homepage url. Apparently the search engines are recognizing these identical pages as separate and duplicate. I know that the standard answer would be to just do a 301 redirect from the long file name to the short file name - end of story. But WT does not allow you to do 301 redirects and they also do not give you the ability to go into the htaccess file to fix this yourself manually. I spoke to the folks at WT tonight and they claim that they automatically do 301 redirects on the platform. But if this true then why am I getting the error message in seomoz? Does anyone know if this is a problem? If so, does anyone here have a fix? Thanks in advance. Sincerely - Bill in Denver
Technical SEO | | anxietycoach0 -
Should I promote each section of my website
Hi, i have a magazine website and i have been heavily promoting the main page of the site thinking that all the work i am doing for the main page which includes links and so on would then pass onto the rest of my site but i have a feeling this is not correct. Can anyone let me know if i should be concentrating on each section of the site and also on my articles should i be promoting these articles or let the search engines pick them up. I already use facebook and twitter to promote new articles but i would like to know if i should be doing more than this
Technical SEO | | ClaireH-1848860 -
Removing some of the indexed pages from my website
I am planning to remove some of the webpages from my website and these webpages are already indexed with search engine. Is there any way by which I need to inform search engine that these pages are no more available.
Technical SEO | | ArtiKalra0