Massive increase in Moz crawl.
-
I have a subdomain which has just started to be crawled by Moz, Previously this wasn't the case.
The sub-domain had 16,000+ issues. Why has Moz started to count sub-domains as part of the main domain, has Google started to do this aswell?
-
Hi Dave!
Thanks for reaching out! My name is Anne, and I helping the Mozzer Alliance today! There could be a few different reasons we've just started crawling this subdomain:
1. Did you recently link this subdomain to your site differently? Our crawler starts with your home page and uses recursive crawling to find pages on your site through the links we find. This could explain why we just started crawling the subdomain.
2. Did you change your robots.txt for this subdomain? Our crawler, Rogerbot, follows robots.txt. If you edited this site recently, this could also explain why we just started crawling your subdomain.
I hope this helps! If you have more questions, please reach out to the help team at https://moz.com/help/contact. From there we can help you troubleshoot what's going on, but keep your campaign details off the interwebz.
Have a great weekend!
Anne
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How get rid of 403 crawl error?
My wordpress website has 162 crawl 403 errors. Based on what I read it means that the server is denying crawlers to access the pages. The pages itself will load so guessing it's just an issue with crawlers only. How do I go about fixing this issue?
On-Page Optimization | | emrekeserr30 -
URL Structure Category Pages -Current Moz Friday-
Hello,
On-Page Optimization | | _Heiko_
regarding #15 of the last moz friday I have a question: http://moz.com/blog/15-seo-best-practices-for-structuring-urls What would you prefer if the lenght of the URL will be still under 60 characters and you have an example like this: Let's call it a specific page in a category. As I like the old shoe examples: You have a page about red shoes in your shoe category. Which URL would you prefer: a) www.mydomain.com/shoes/red-shoe b) www.mydomain.com/shoes/red Personally I would prefer a) or would you already consider this as spammy? My real example is not that trivial like the shoe example and the categories will be in plural and the specific pages always in singular (like in the example shoes vs shoe). c) would be to put it independently from the side structure on www.mydomain.com/red-shoe - but personally I have the experience that a) or b) will help the rankings of the category page if you have the specific pages in the same subfolder. What's your opinion on this?1 -
404 errors in wordpress... Pages have never existed so why is google trying to crawl them?
I've just logged into webmaster tools and have over 100 404 errors. I'm running wordpress and I recently added child pages to 2 of my categories like so. www.mydomain.com/category1/lincolnshire www.mydomain.com/category1/cambridgeshire etc... The 404 errors though are for pages or categories I've never created though. I have over 20 root categories but decided to test adding child pages to only two of them. The 404 errors are for www.mydomain.com/category5/cambridgeshire .... It seem that gogle has tried to crawl these pages that don't exist. Can anyone explain what's going on? When I click 'linked from' in webmaster tools it's showing links from pages on my site that don't exist also.
On-Page Optimization | | SamCUK0 -
Why so many crawl errors?
Our site is showing it has a ton of crawl errors in the back end, mostly concerning duplicate content within our blog. The content is unique however. We know this for certain because it's done in house or put together by some of the freelance writers we work with. The site is for an RV dealership and we're using a template-based system from a well known company. Any ideas on what may be causing this?
On-Page Optimization | | BlakeArbogast0 -
What's the best way to handle crawling of photo gallery?
When you have a photo gallery with many search filters and loads and loads of pages, is it best to block all the filters and use google's pagination code? Ex: http://photo.net/gallery/photocritique/filter This site has pages for many different queries. While the page titles are unique, the pages are showing duplicated content.
On-Page Optimization | | cakelady0 -
Crawl Diagnostics not working?
i've been in the crawl diagnostics of my website. I only have 1 page crawled and no errors. Last Crawl Completed: Jul. 12th, 2012 Next Crawl Starts: Jul. 19th, 2012 Do you have an idea of how to fiw it? Thanks a lot
On-Page Optimization | | Ericc220 -
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them.
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them. http://pro.seomoz.org/campaigns/95663/issues#notice-issues
On-Page Optimization | | cyaindc0 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0