Sub Domains and Robot.txt files...
-
This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left.
I have a sub level domain on which a website sits. The Main domain has a robots.txt file that disallows all robots. It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website. My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain? I wouldn't have thought so but I can find nothing else.
Thanks in advance.
-
Thank you, Mr. Young. I believed this to be the case (that it wasn't the robots.txt file) but I could think of nothing else. I have since been indexed.
-
The way that Google finds robots.txt files is by taking your URL, and adding /robots.txt to it. So a good way to see if the robots.txt file is affecting your subdomain is to go to subdomain.domain.com/robots.txt. If the file exists, then it is affecting your subdomain. If it doesn't, then it's only active on your main domain.
Getting indexed is function of having unique content and pagerank, so make sure your subdomain has unique content and links if you're having trouble getting it indexed. Submitting a sitemap is no guarantee that Google will index your site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multisite domain
good morning I have a wordpress site I have activated the multisite, currently the site has a domain authority of 8, when I publish a post, it is indexed quite quickly, if I publish a post in a language other than the /es subdomain it takes 24 hours why? If the author domain is the same, why does the employee take longer to be indexed on Google? Thank you
Technical SEO | | alainscilly770 -
Blocking in Robots.txt and the re-indexing - DA effects?
I have two good high level DA sites that target the US (.com) and UK (.co.uk). The .com ranks well but is dormant from a commercial aspect - the .co.uk is the commercial focus and gets great traffic. Issue is the .com ranks for brand in the UK - I want the .co.uk to rank for brand in the UK. I can't 301 the .com as it will be used again in the near future. I want to block the .com in Robots.txt with a view to un-block it again when I need it. I don't think the DA would be affected as the links stay and the sites live (just not indexed) so when I unblock it should be fine - HOWEVER - my query is things like organic CTR data that Google records and other factors won't contribute to its value. Has anyone ever blocked and un-blocked and whats the affects pls? All answers greatly received - cheers GB
Technical SEO | | Bush_JSM0 -
Robots.txt - "File does not appear to be valid"
Good afternoon Mozzers! I've got a weird problem with one of the sites I'm dealing with. For some reason, one of the developers changed the robots.txt file to disavow every site on the page - not a wise move! To rectify this, we uploaded the new robots.txt file to the domain's root as per Webmaster Tool's instructions. The live file is: User-agent: * (http://www.savistobathrooms.co.uk/robots.txt) I've submitted the new file in Webmaster Tools and it's pulling it through correctly in the editor. However, Webmaster Tools is not happy with it, for some reason. I've attached an image of the error. Does anyone have any ideas? I'm managing another site with the exact same robots.txt file and there are no issues. Cheers, Lewis FNcK2YQ
Technical SEO | | PeaSoupDigital0 -
Spam flags for sub-domain
Hi Moz Community, I am reviewing our website via MOZ and found the following issues: http://shopwindowcleaningresource.com/ 1. No Contact (please refer to the attached image). However, we have an up to date contact info on our website. We have social buttons at the footer, our telephone number is at the top and we have a contact us page. Any idea, why we are being rated as such and how to resolve it? Any help will be greatly appreciated, thanks in advance. 327M9Fc
Technical SEO | | Shop-Sq0 -
Domain taken. Which is better? Using hypens or longer domain.
I am wanting to set up an e commerce site and the domain name that I want is taken. I am considering using a domain that has the main keyword I want to rank for as the domain. I have heard chatter of google penalizing these types of sites and it seems that it hasn't come about. This is something that I would like to test out. So if "electricscooters.com" is taken, should I use "electric-scooters.com" or "electricscooters4less.com" Just wondering if the hyphenated or the longer domain will rank higher. The site won't be spammy at all, I will carry a few different companies that offer similar products. So for this case, I would only sell scooters from a few different manufacturers. Feedback would be appreciated!
Technical SEO | | Dave_Whitty0 -
I think I have a penalty on my domain...
my domain is www.brighttights.com it is an affiliate marketing website in the niche of tights and lingerie. A few months back my traffic was pretty good, doing about 500 hits a day from product search terms only. After the panda updates I blocked all the product pages from google as they were duplicate content and I am now working on a program of seing for the category and homepages instead. I am using much more generic, and high volume, keywords for these. Several months later I seem to not only be down to 7 people a day on my website but i'm not even ranking for terms such as "bright tights". I used to be no1 for this. I have domain authority of 27 so it's not terrible, competitors on the first page range from 45 to 9. This lack of ranking for the sites name/domain name term is leading me to wonder if I have a penalty on the site. Any feedback would be gratefully received.
Technical SEO | | Grumpy_Carl0 -
Site not being Indexed that fast anymore, Is something wrong with this Robots.txt
My wordpress site's robots.txt used to be this: User-agent: * Disallow: Sitemap: http://www.domainame.com/sitemap.xml.gz I also have all in one SEO installed and other than posts, tags are also index,follow on my site. My new posts used to appear on google in seconds after publishing. I changed the robots.txt to following and now post indexing takes hours. Is there something wrong with this robots.txt? User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /wp-login.php Disallow: /wp-login.php Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /author Disallow: /category Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /login/ Disallow: /wget/ Disallow: /httpd/ Disallow: /*.php$ Disallow: /? Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.cgi$ Disallow: /*.xhtml$ Disallow: /? Disallow: /*?Allow: /wp-content/uploads User-agent: TechnoratiBot/8.1 Disallow: ia_archiverUser-agent: ia_archiver Disallow: / disable duggmirror User-agent: duggmirror Disallow: / allow google image bot to search all imagesUser-agent: Googlebot-Image Disallow: /wp-includes/ Allow: /* # allow adsense bot on entire siteUser-agent: Mediapartners-Google* Disallow: Allow: /* Sitemap: http://www.domainname.com/sitemap.xml.gz
Technical SEO | | ideas1230 -
Buying a new domain
Hello guys! We are in process of buying a new domain. How can we be sure that this domain is not blacklisted and are there any steps to take in order to be sure that whatever we are buying is actually in "good shape"? Thanks much!
Technical SEO | | echo10