Robots.txt help
-
Hi Moz Community,
Google is indexing some developer pages from a previous website where I currently work:
ddcblog.dev.examplewebsite.com/categories/sub-categories
Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?
As always, your expertise is greatly appreciated,
-Reed
-
The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
-
Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
-
You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sudden Rankings Drop for Good Keywords.. Did I Do This? Please Help :(
Hello, I noticed a gradual rankings drop for 3 important keywords over the last month, with a pretty big plummet the last two weeks. Overall in the last month+ we dropped from position 9 to 41.I noticed this when I dug further after noticing traffic dropping since February (not a drastic traffic drop). I should note that the keywords took people to my client's homepage. Their branded keywords have no suffered and I looked at a couple others that haven't either. Now, there is a link in the site footer (we have site wide header and footer) that takes you to a static page that contains links for the 2 digital flipbook catalogs the customer has (one for US and one for Canada). My concern is that at the end of January I had a developer implement a noindex/nofollow meta robot & robots.txt disallow specifically on the HTML pages/URL of the Canadian catalog ONLY. It specifically pointed to that flipbook URL. This catalog is nearly identical to the US catalog and I thought I'd be eliminating duplicate content and helping with crawl budget. After looking further into it last week (reading up about internal nofollows not necessarily being detrimental, but not recommended) and noticing the drop in search visibility traffic (starting gradually in March), I had the disallow/nofollow removed. This was last week, and over this last week the traffic took an even bigger drop (not amazingly drastic but enough to be concerned) and I noticed the keywords that we did ok for dropped even more this last week (down to 41). I'm concerned this has to do with the change I made at the end of January and reversed back. I should note that I don't think these catalogs or the static page that links to them brought any traffic. The keywords I am concerned about fell on our homepage (where the link to the static page that contains the links to both catalogs is in the sitewide footer) The catalogs are a couple hundred pages. I honestly don't see how this could do it, unless it has something to do with the footer being sitewide? There have been site upgrades/dev changes over the last couple months too (although I am not sure if that affected other clients who received the same upgrade), so this is hard to pinpoint. Sorry this is so long but I'd appreciate someone offering some insight to help ease my mind a bit!
Intermediate & Advanced SEO | | AliMac260 -
Moving blog to a subdomain, how can I help it rank?
Hi all, We recently moved our blog to a sub-domain where it is hosted on Wordpress. It was very recent and we're actively working on the SEO, but any pointers on getting the subdomain to rank higher than the old blog posts would be terrific. Thanks!
Intermediate & Advanced SEO | | DigitalMoz0 -
Need help with huge spike in duplicate content and page title errors.
Hi Mozzers, I come asking for help. I've had a client who's reported a staggering increase in errors of over 18,000! The errors include duplicate content and page titles. I think I've found the culprit and it's the News & Events calender on the following page: http://www.newmanshs.wa.edu.au/news-events/events/07-2013 Essentially each day of the week is an individual link, and events stretching over a few days get reported as duplicate content. Do you have any ideas how to fix this issue? Any help is much appreciated. Cheers
Intermediate & Advanced SEO | | bamcreative0 -
Recovering from robots.txt error
Hello, A client of mine is going through a bit of a crisis. A developer (at their end) added Disallow: / to the robots.txt file. Luckily the SEOMoz crawl ran a couple of days after this happened and alerted me to the error. The robots.txt file was quickly updated but the client has found the vast majority of their rankings have gone. It took a further 5 days for GWMT to file that the robots.txt file had been updated and since then we have "Fetched as Google" and "Submitted URL and linked pages" in GWMT. In GWMT it is still showing that that vast majority of pages are blocked in the "Blocked URLs" section, although the robots.txt file below it is now ok. I guess what I want to ask is: What else is there that we can do to recover these rankings quickly? What time scales can we expect for recovery? More importantly has anyone had any experience with this sort of situation and is full recovery normal? Thanks in advance!
Intermediate & Advanced SEO | | RikkiD220 -
301 redirect on Windows IIS. HELP!
Hi My six-year-old domain has always existed in four forms: http://www**.**mydomain.com/index.html http://mydomain.com/index.html http://mydomain.com/ http://www.mydomain.com My webmaster claims it’s “impossible” to do a 301 redirect from the first three to the fourth. I need simple instructions to guide him. The site’s hosted on Windows running IIS Here’s his rationale: These are all the same page, so they can’t redirect to themselves. Index.html is the default page that loads automatically if you don’t specify a page. If I put a redirect into index.html it would just run an infinite redirect loop. As you can see from the IIS set up, both www.mydomain and mydomain.com point to the same location ( VIEW IMAGE HERE ) _Both of these use index.html as the default document ( VIEW IMAGE 2 HERE ) _
Intermediate & Advanced SEO | | Jeepster0 -
Duplicate Page Content / Titles Help
Hi guys, My SEOmoz crawl diagnostics throw up thousands of Dup Page Content / Title errors which are mostly from the forum attached to my website. In-particular it's the forum user's profiles that are causing the issue, below is a sample of the URLs that are being penalised: http://www.mywebsite.com/subfolder/myforum/pop_profile.asp?mode=display&id=1308 I thought that by adding - http://www.mywebsite.com/subfolder/myforum/pop_profile.asp to my robots.txt file under 'Ignore' would cause the bots to overlook the thousands of profile pages but the latest SEOmoz crawl still picks them up. My question is, how can I get the bots to ignore these profile pages (they don't contain any useful content) and how much will this be affecting my rankings (bearing in mind I have thousands of errors for dup content and dup page titles). Thanks guys Gareth
Intermediate & Advanced SEO | | gaz33420 -
Will blocking urls in robots.txt void out any backlink benefits? - I'll explain...
Ok... So I add tracking parameters to some of my social media campaigns but block those parameters via robots.txt. This helps avoid duplicate content issues (Yes, I do also have correct canonical tags added)... but my question is -- Does this cause me to miss out on any backlink magic coming my way from these articles, posts or links? Example url: www.mysite.com/subject/?tracking-info-goes-here-1234 Canonical tag is: www.mysite.com/subject/ I'm blocking anything with "?tracking-info-goes-here" via robots.txt The url with the tracking info of course IS NOT indexed in Google but IT IS indexed without the tracking parameters. What are your thoughts? Should I nix the robots.txt stuff since I already have the canonical tag in place? Do you think I'm getting the backlink "juice" from all the links with the tracking parameter? What would you do? Why? Are you sure? 🙂
Intermediate & Advanced SEO | | AubieJon0 -
Will links to a subdomain help it rank?
I have an affiliate subdomain on a larger company's domain. (For example I have: www.victor.company.com on www.company.com). Would working to attain backlinks to the subdomain help it rank or will I just be putting forth my effort and helping the domain rank?
Intermediate & Advanced SEO | | VictorVC0