SEOMOZ crawler is still crawling a subdomain despite disallow
-
This is for our client with a subdomain. We only want to analyze their main website as this is the one we want to SEO. The subdomain is not optimized so we know it's bound to have lots of errors.
We added the disallow code when we started and it was working fine. We only saw the errors for the main domain and we were able to fix them. However, just a month ago, the errors and warnings spiked up and the errors we saw were for the subdomain.
As far as our web guys are concerned. the disallow code is still there and was not touched. User-agent: rogerbot Disallow: /
We would like to know if there's anything we might have unintentionally changed or something we need to do so that the SEOMOZ crawler will stop going through the subdomain.
Any help is greatly appreciated!
-
Thanks Peter for your assistance.
Hope to hear from the SEOMOZ team soon with regards to this issue.
-
John,
Thanks for writing in! I would like to take a look at which project you guys were working with that this is happening. I will go ahead and start a ticket so we can better answer your questions
You should hear from me soon!Best,
Peter
Moz Help Team. -
I have heard of this before recently, I think possibly the moz crawler all or sometimes now just ignores the disallow because it is not a usual S.E crawler.
Hopefully one of the staff can provide some insight in this for you.
All the best.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Paginated category pages still showing in Google
Despite our blog using rel=next and rel=”prev” we’re still finding paginated pages getting impressions in Google, suggesting they are taking up unnecessary crawl budget. An example is: https://www.theukdomain.uk/seo/page/2/ What steps would you recommend I take to most benefit my sites SEO? Thanks, Sam
Intermediate & Advanced SEO | | sjefferies0 -
Still Seeing GSC Traffic in HTTP Property Post-Migration
We migrated to HTTPS in June 2017, so why would I still be seeing a bit of traffic in our HTTP property in Google Search Console? QyqQ2
Intermediate & Advanced SEO | | catbur0 -
We killed our SEO, but how come some of our keywords are still in the top 1-3
I am looking for the answer for this interesting question: 1. I have a static page with NO information on it, this is almost completelyblank, only a search box on it, which does nothing! The information it contains is absolutely zero, but the page has got a specific URL which is = the keyword i will look for 2. And i have another page which is fully optimized with the help of on page grader (97%) for anoter specific keyword Q, in case of 1: Searching in google for a keyword which is = the page www.domain.hu/keyword, and i will have top 1-3 serp!???? (i say it again, the page contains no information...and the keyword is really frequent, in google adwords it says that this keyword which has got high competition) in case of 2. i have the /url which is completely the same as the keyword, have 97% on page grade, and i see that week by week i can only move upwards a little in the serp. Created unique content a lot, made several changes on this page and like no position changing. So the question is WHY in case 1 i can be with no information in (empty static page) top 1-3 for a really hard keyword, and why i cannot move upward on the list for a not so frequent keyword however i did everything i could??
Intermediate & Advanced SEO | | Neckermann0 -
Third Party Subdomain Slow Load Times Affect Our SERPs?
We have a third party subdomain that is not hosted on our server (smugmug.com gallery). It periodically has slow load times. The question is; does anybody know if the SERPs, or more specifically Google, would see that subdomain as our site? I want to gather insight into whether or not this might affect our results. Thank you!
Intermediate & Advanced SEO | | leslieevarts0 -
How to handle a blog subdomain on the main sitemap and robots file?
Hi, I have some confusion about how our blog subdomain is handled in our sitemap. We have our main website, example.com, and our blog, blog.example.com. Should we list the blog subdomain URL in our main sitemap? In other words, is listing a subdomain allowed in the root sitemap? What does the final structure look like in terms of the sitemap and robots file? Specifically: **example.com/sitemap.xml ** would I include a link to our blog subdomain (blog.example.com)? example.com/robots.xml would I include a link to BOTH our main sitemap and blog sitemap? blog.example.com/sitemap.xml would I include a link to our main website URL (even though it's not a subdomain)? blog.example.com/robots.xml does a subdomain need its own robots file? I'm a technical SEO and understand the mechanics of much of on-page SEO.... but for some reason I never found an answer to this specific question and I am wondering how the pros do it. I appreciate your help with this.
Intermediate & Advanced SEO | | seo.owl0 -
SubDomain vs. SubFolder
I know this subject has been discussed many, many times before. But it is now 2013, and Google continues to tweak and change their algo to build upon the best delivered results for users. So the questions are: Does Google still treat subdomains as a completely separate and unique domain from the root? If so, is it a good SEO strategy to split up, when it fits, a website into subdomains with links pointing back to the root or main domain? As a company we have several subdomains with some of our categories. For example our main site is www.iboats.com. This site has all our boat products. But we set up subdomains several years ago for the following: boatcovers.iboats.com boatpropellers.iboats.com biminitops.iboats.com And we have our fourms as a subdomain: forums.iboats.com Splitting them out were originally done for SEO reasons, but now is more for better managing our main categories. It appears that Google is treating our subdomains as part of our main root domain anyway, so I don't see the SEO value anymore. If we were to move the subdomains into subfolders of the root, I'm wondering if we might see a boost in SEO value having more pages within the main website? I'd be interested in everyone's thoughts on this subject.
Intermediate & Advanced SEO | | tdawson090 -
Subdomain or New Domain or Directory path?
Hi Mozzers, I have a dilemma here. I have one of my clients that has its American website and would like to promote its business in canada(2 locations only vs more than 40 in the u.s). What would be the best SEO decision here for the 2 Canadian locations: should i go for a new root domain? if yes why (www.example.ca) a new subdomain? if yes why (ca.example.com) or just following a directory path solution under the american site? (www.example.com/canadiancity) Thanks Ty
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
What is the best tool to crawl a site with millions of pages?
I want to crawl a site that has so many pages that Xenu and Screaming Frog keep crashing at some point after 200,000 pages. What tools will allow me to crawl a site with millions of pages without crashing?
Intermediate & Advanced SEO | | iCrossing_UK0