Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Only Images Show Up in Log Files
Has anyone ever seen a log file analysis return only images and no actual page URLs?
Technical SEO | | LoganRay0 -
Clarification regarding robots.txt protocol
Hi,
Technical SEO | | nlogix
I have a website , and having 1000 above url and all the url already got indexed in Google . Now am going to stop all the available services in my website and removed all the landing pages from website. Now only home page available . So i need to remove all the indexed urls from Google . I have already used robots txt protocol for removing url. i guess it is not a good method for adding bulk amount of urls (nearly 1000) in robots.txt . So just wanted to know is there any other method for removing indexed urls.
Please advice.0 -
How to get only the most needed css for faster loading?
I have been using the Firefox duster app to clean up my css so only the page rendering css is loaded when my page is loaded. But it doesn't seem to be working now. Does anyone know of another tool that will do this for me?
Technical SEO | | RoxBrock0 -
When to file a Reconsideration Request
Hi all, I don't have any manual penalties from Google but do have a unnatural links message from them back in 2012. We have removed some of the spammy links over the last 2 years but we're now making a further effort and will use the disavow tool once we've done this. Will this be enough once I submit the file or should I / can I submit a Reconsideration Request as well? Do I have to have a manual penalty item in my webmaster account to be able to submit a request? Thanks everyone!
Technical SEO | | KerryK0 -
Google indexing despite robots.txt block
Hi This subdomain has about 4'000 URLs indexed in Google, although it's blocked via robots.txt: https://www.google.com/search?safe=off&q=site%3Awww1.swisscom.ch&oq=site%3Awww1.swisscom.ch This has been the case for almost a year now, and it does not look like Google tends to respect the blocking in http://www1.swisscom.ch/robots.txt Any clues why this is or what I could do to resolve it? Thanks!
Technical SEO | | zeepartner0 -
1000+ links from domain to subdomain - impacting?
Hi there! I'm wondering if anyone could kindly help me with this question. I have a domain let's say: thisdomain.com - and I also have a subdomain called widgets.thisdomain.com thisdomain.com has approx 2000 pages on it. There are site wide links in the top nav, side nav and footer pointing to widgets.thisdomain.com (all with the same anchor text - not branded - it's a money term). My question is, do you think this would be causing some kind of algorithmic penalty (over optimisation / penguin)? Or, do you think Google realises these links are pointing to a subdomain and says, "hey, OK we understand these two domains could be related" or given subdomains are seen as essentially standalone websites, the algo is saying "hang on a minute, pal, this looks fishy, why are you linking to this subdomain 1000+ times with sitewide links on the same anchor text"? Should I nofollow these sitewide links? Do you think that will help? I would really appreciate any help on this one. Cheers!
Technical SEO | | WCR0 -
Do I need an XML sitemap?
I have an established website that ranks well in Google. However, I have just noticed that no xml sitemap has been registered in Google webmaster tools, so the likelihood is that it hasn't been registered with the other search engines. However, there is an html sitemap listed on the website. Seeing as the website is already ranking well, do I still need to generate and submit an XML sitemap? Could there be any detriment to current rankings in doing so?
Technical SEO | | pugh0 -
I have been Hit with UNWANTED Links and I need to know how to remove them?
I have been looking at Google Webmasters Tools and saw that suddenly I have this huge amount of links form an unknown source “SiteLoki” never heard of it? Didn’t want it! Need to get rid of it!
Technical SEO | | rdominey
My Website www.getyourphotosoncanvas.com experienced a sudden and dramatic loss of presence on Google about three weeks ago. I first thought it was something to do with the new Google Algorithm. I also contacted Google to be re-evaluated. They replied:
Reconsideration request for http://www.getyourphotosoncanvas.com/: No manual spam actions foundMay 10, 2012 Dear site owner or webmaster of http://www.getyourphotosoncanvas.com/,
We received a request from a site owner to reconsider http://www.getyourphotosoncanvas.com/ for compliance with Google's Webmaster Guidelines. - - - - - -
We reviewed your site and found no manual actions by the webspam team that might affect your site's ranking in Google. There's no need to file a reconsideration request for your site, because any ranking issues you may be experiencing are not related to a manual action taken by the webspam team.
Google Search Quality Team I have ran all kinds of web crawl tests, Google webmaster, talked with SEO “Experts” and still can not figure out what is happening. I decided to use a couple of SEOmoz tools to try to help me explain what is happening. I figured that if I could take a very specific and unique KeyPhrase and run it on a specific page that I might be able to better explain what is happening. Basically, We appear to be no longer searchable by key words or phrases on google?
There was another post regarding canonical tag issues; resolved, I removed all canonical tags for now. Today I discovered the links and I suspect that having even
1 link for this website will penalize you in Google. Here are some Webmaster Tools pages for reference: I also noted that the image for the link is from an old site hosted on www.Globat.com. I don't know what that significance might be. I left them in January. Thanks in advance for any help and advice you can offer, Ray Webmaster-Tools-Query_Your-Photo-On-Canvas.jpg Webmaster-Tools-Query_Photo-On-Canvas.jpg Webmaster-Tools-Query_Photos-On-Canvas.jpg Webmaster-Tools-Links_siteloki.jpg Webmaster-Tools-Links_siteloki_links.jpg0