SEOMOZ crawler is still crawling a subdomain despite disallow
-
This is for our client with a subdomain. We only want to analyze their main website as this is the one we want to SEO. The subdomain is not optimized so we know it's bound to have lots of errors.
We added the disallow code when we started and it was working fine. We only saw the errors for the main domain and we were able to fix them. However, just a month ago, the errors and warnings spiked up and the errors we saw were for the subdomain.
As far as our web guys are concerned. the disallow code is still there and was not touched. User-agent: rogerbot Disallow: /
We would like to know if there's anything we might have unintentionally changed or something we need to do so that the SEOMOZ crawler will stop going through the subdomain.
Any help is greatly appreciated!
-
Thanks Peter for your assistance.
Hope to hear from the SEOMOZ team soon with regards to this issue.
-
John,
Thanks for writing in! I would like to take a look at which project you guys were working with that this is happening. I will go ahead and start a ticket so we can better answer your questions
You should hear from me soon!Best,
Peter
Moz Help Team. -
I have heard of this before recently, I think possibly the moz crawler all or sometimes now just ignores the disallow because it is not a usual S.E crawler.
Hopefully one of the staff can provide some insight in this for you.
All the best.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Paginated category pages still showing in Google
Despite our blog using rel=next and rel=”prev” we’re still finding paginated pages getting impressions in Google, suggesting they are taking up unnecessary crawl budget. An example is: https://www.theukdomain.uk/seo/page/2/ What steps would you recommend I take to most benefit my sites SEO? Thanks, Sam
Intermediate & Advanced SEO | | sjefferies0 -
GG webmaster tool crawl rate
Hello, I just redid my website and suddenly saw a spike in the crawl rate for about 2 weeks and I am now back to where I was on average. Is it normal ? My guess it that this increase was due to the change in site and new links ? However,I just want to make sure it is normal that it is back to a "normal rate" now that it has discovered all the links. Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
First Link on Page Still Only Link on Page?
Bruce Clay and others did some research and found that the first link on the page is the most important and what is accredited as the link. Any other links on the page mean nothing. Is this still true? And in that case, on an ecommerce site with category links in the top navigation (which is high on the code), is it not useful to link to categories in the content of the page? Because the category is already linked to on that page. Thank you, Tyler
Intermediate & Advanced SEO | | tylerfraser0 -
Do search engines crawl links on 404 pages?
I'm currently in the process of redesigning my site's 404 page. I know there's all sorts of best practices from UX standpoint but what about search engines? Since these pages are roadblocks in the crawl process, I was wondering if there's a way to help the search engine continue its crawl. Does putting links to "recent posts" or something along those lines allow the bot to continue on its way or does the crawl stop at that point because the 404 HTTP status code is thrown in the header response?
Intermediate & Advanced SEO | | brad-causes0 -
SubDomain vs. SubFolder
I know this subject has been discussed many, many times before. But it is now 2013, and Google continues to tweak and change their algo to build upon the best delivered results for users. So the questions are: Does Google still treat subdomains as a completely separate and unique domain from the root? If so, is it a good SEO strategy to split up, when it fits, a website into subdomains with links pointing back to the root or main domain? As a company we have several subdomains with some of our categories. For example our main site is www.iboats.com. This site has all our boat products. But we set up subdomains several years ago for the following: boatcovers.iboats.com boatpropellers.iboats.com biminitops.iboats.com And we have our fourms as a subdomain: forums.iboats.com Splitting them out were originally done for SEO reasons, but now is more for better managing our main categories. It appears that Google is treating our subdomains as part of our main root domain anyway, so I don't see the SEO value anymore. If we were to move the subdomains into subfolders of the root, I'm wondering if we might see a boost in SEO value having more pages within the main website? I'd be interested in everyone's thoughts on this subject.
Intermediate & Advanced SEO | | tdawson090 -
Flat vs. subdomain web structure
I am building a site which sells a product in 50 states and in each state we will have independt partners. From an SEO perspective, what are the tradeoffs in using a single domain vs. having each state a subdomain? Each state also has varying regulatory issues that are specific to that state.
Intermediate & Advanced SEO | | uwaim20120 -
Is it possible to 301 re-direct an entire subfolder on a subdomain?
So, for example, subdomain.domain/folder subdomain.domain/folder2 Is it possible to set up a 301-re-direct for an entire folder on a subdomain? If so, how? What is the correct code for this?
Intermediate & Advanced SEO | | nicole.healthline0 -
Google places: 7 weeks and still not verified. Reasons?
I've created Google Places entries for the business' 25 locations in 8 countries. It's been 7 weeks and Google hasn't validated my bulk upload yet. Which of the following would you say might do the trick? URL: I have pointed to http://site.com/country/city for every location. Should I point to http://site? I've created categories in English, but being this about local I guess I should do it in the local language of each country... Should I cancel the bulk upload and do one country per country? My list of countries include Latin America, Europe, Africa and Asia so I don't know if manual verification is done by google centrally and that might be a problem. Otherwise the entries are (IMHO) well written, detailed and all. But I'm kind of desperate now because it's 7 weeks already... thanks for you help
Intermediate & Advanced SEO | | TIBA0