Moz crawls
-
Hi all! I'm running two separate campaign crawls at the moment that I'm having a few issues with. The first is http://www.muchbetteradventures.com/.
I'm tracking this at root to try an pick any problems with a previous version of the site, which is on the v1.muchbetteradventures subdomain. However, I'm only seeing 100 or so pages as being crawled in Moz, compared to thousands in Google. There's no access blocked alerts in Moz either.
The second is http://www.sothebys.com/. I started this crawl at root a few days ago and no pages at all have been processed. Very odd!
Any advice would be much appreciated.
-
Thanks Reuben! Just getting my head around Screaming Frog! Can it be used to flag up duplicate content issues like Moz?
-
Hello and great question! If you are ever missing pages from your crawl, you can always email help@moz.com with the name of the campaign and a couple examples of missing pages. We can usually diagnose this issue pretty quickly. In a lot of cases, this could be due to the way some domains are forwarded or redirected. I encourage you to email us the issue with a few examples so that the Help Team, including myself, can dig in and figure out whats going on. Sorry for the delay, I hope we can help get to the bottom of your crawl issues.
-
1. Google's index may or may not be reflective of the pages you actually have on your site—likely the latter, hence the difference between the pages on your site and the pages in Google's index. Right now, the v1 subdomain times out, so no crawlers will be able to do anything with it. The www domain only has about 250 or so URLs.
2. Sotheby's website blocks all but a few crawlers, which is why Moz would not crawl any pages. http://www.sothebys.com/robots.txt
I would suggest using your own crawling software to untangle these issues. If you aren't familiar with Screaming Frog, you should check it out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Performance Metrics crawl error
I am getting an error:
Product Support | | bhsiao 0
Crawl Error for mobile & desktop page crawl - The page returned a 4xx; Lighthouse could not analyze this page.
I have Lighthouse whitelisted, is there any other site I need to whitelist? Anything else I need to do in Cloudflare or Datadome to allow this tool to work?1 -
Check out our Moz Status Page for current outages
You can see updates regarding known outages on our Moz Status page here: https://moz.com/help/status/maintenance-and-outages
Product Support | | HayleyBowyer0 -
Crawl test
I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?
Product Support | | Karen_Dauncey0 -
How to use moz to find spam backlinks
I am new to MOZ pro. I have lots of spam backlinks from a company that blackhatted me about 5 years ago. Can MOZ list these for me? I don't know how. Thanks
Product Support | | TomHW0 -
Link to Moz Pro home 404s, and then I stumbled upon Moz's staging site...
I'm wary of giving more information as it's probably best that little to no one be able to access Moz staging for one reason or another. I should also mention that this was done very unintentionally on my part. That said, from the 404 from Moz Pro home, I tried to access "My QA", and that's when I realized that everything I did thereafter was within staging.moz.com (and posted a question similar to this one). Does Moz permit access to their staging site, or did I stumble upon a mistake - or Moz-stake, if you will?
Product Support | | Lumina0 -
How to find first set of Moz rankings?
I'm trying to compare the state of our seo before we started moz to now. I can't find the first set of rankings and on-page grades when I initially setup Moz. I do have access to our first report, but in that first week, I made a number of changes based on the initial crawl/rankings/on-page grades. Anyone know how to get that? Thanks, Ruben
Product Support | | KempRugeLawGroup0 -
Email address for moz.com?
I signed up for a trial on seomoz and today i found out moz.com has charged me for a subscription without my knowledge. how can I contact them?
Product Support | | iareef0