Sub Domains and Robot.txt files...
-
This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left.
I have a sub level domain on which a website sits. The Main domain has a robots.txt file that disallows all robots. It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website. My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain? I wouldn't have thought so but I can find nothing else.
Thanks in advance.
-
Thank you, Mr. Young. I believed this to be the case (that it wasn't the robots.txt file) but I could think of nothing else. I have since been indexed.
-
The way that Google finds robots.txt files is by taking your URL, and adding /robots.txt to it. So a good way to see if the robots.txt file is affecting your subdomain is to go to subdomain.domain.com/robots.txt. If the file exists, then it is affecting your subdomain. If it doesn't, then it's only active on your main domain.
Getting indexed is function of having unique content and pagerank, so make sure your subdomain has unique content and links if you're having trouble getting it indexed. Submitting a sitemap is no guarantee that Google will index your site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New domain wipes out domain authority
A client wanted to change their domain name, which we have now done. The site content itself is exactly the same. We put 301 redirect links in so that Google searchers would redirect from the old site to the new one. However Moz then said that it couldn't crawl the old domain because of the redirects and advised creating a brand new campaign for the new domain. We have done this but now Moz says that the domain authority of the new site is 2 (it was 14 on the old domain). Specifics are:
Technical SEO | | mfrgolfgti
old domain: https://ryemeadcleaning.co.uk
new domain: https://ryemeadgroup.co.uk So basically it seems like we're starting again from scratch with the new domain and all the SEO from the old domain has been lost? Have we done it wrong?0 -
Robots.txt on subdomains
Hi guys! I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?
Technical SEO | | Whittie0 -
Followed Linking Root Domains and No Followed Linking Domains
If you have more NoFollowed Linking Root Domains than Followed Linking Root Domains is that a problem?
Technical SEO | | INN0 -
No indexing url including query string with Robots txt
Dear all, how can I block url/pages with query strings like page.html?dir=asc&order=name with robots txt? Thanks!
Technical SEO | | HMK-NL0 -
Root domain ranks higher than sub pages
Our website is over 10 years old and has been very successful with google. We rank highly for our keywords, but recently a strange thing has happend. Google has indexed a video from our front page (which no longer exists) and shows this listing instead of the sub pages. This means the listing is irrelevant to the search term. Term is "Junior Cricket Bats"
Technical SEO | | NickKer
https://www.google.co.uk/search?sourceid=navclient&ie=UTF-8&rlz=1T4GGLL_en-GBGB377GB377&q=junior+cricket+bats should some up with this page
http://www.cricketsupplies.com/junior-cricket-bats.asp but shows the root domain
http://www.cricketsupplies.com/ with this video link which does not exist We should be ranking really well , and were , but now not so good , even though SEOmoz loves our pages if anybody can see anything obvious i would be thanksful. regards0 -
Any way around buying hosting for an old domain to 301 redirect to a new domain?
Howdy. I have just read this QA thread, so I think I have my answer. But I'm going to ask anyway! Basically DomainA.com is being retired, and DomainB.com is going to be launched. We're going to have to redirect numerous URLs from DomainA.com to DomainB.com. I think the way to go about this is to continue paying for hosting for DomainA.com, serving a .htaccess from that hosting account, and then hosting DomainB.com separately. Anybody know of a way to avoid paying for hosting a .htaccess file on DomainA.com? Thanks!
Technical SEO | | SamTurri0 -
Mobile site: robots.txt best practices
If there are canonical tags pointing to the web version of each mobile page, what should a robots.txt file for a mobile site have?
Technical SEO | | bonnierSEO0 -
Two spelling of a domain
I have a customer with two spellings of their domain name. I set up an account for spelling A and forwarded all the email boxes to spelling B becuase people tend to remember spelling A more of the time. Spelling B is the real web site. I also want any www. traffic for spelling A to go to spelling B so I used this .htaccess file in the root of spelling A Options +Indexes +FollowSymLinks
Technical SEO | | freestone
RewriteEngine On
RewriteRule ^(.*)$ http://www.B.com/$1 [R=301,L] I use to just forward A to B from the registrar but made this change to allow for email spelled either way. My question is does this create a duplicate site issue for the bots? Is this in anyway an SEO negative and if so is there a better way to do this. Thanks jw0