Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt help

Intermediate & Advanced SEO

566

IceIcebaby last edited by

Hi Moz Community,

Google is indexing some developer pages from a previous website where I currently work:

ddcblog.dev.examplewebsite.com/categories/sub-categories

Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?

As always, your expertise is greatly appreciated,

-Reed
1 Reply Last reply
Reply Quote 0
KeriMorgret @LesleyPaone last edited by

The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
1 Reply Last reply
Reply Quote 1
LesleyPaone last edited by

Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
1 Reply Last reply
Reply Quote 1
KeriMorgret last edited by

You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Help with structure for optimizing Photography Website SEO

Hey guys , I am building a photography website and currently I have it setup the following way for my image galleries : https://ricfrancophotography.com/portfolio/norway-landscape-photography/#!gallery6618-6577 This provides me with an individual link for each of the images in my photography gallery but as a gallery these have obviously no content and I figured the best way would be to add the images I want to work my SEO for to individual blog posts . So here is what I did so far : - Added a link to the caption of each image inside the lightbox that is linking to the individual blog post - In order to not break the navigation I made the post with the content for each image open in a modal popup (it changes the link in the top bar but once closed it goes back to the gallery) . - I made the image inside the post link back to the fullsize image in the lightbox gallery when clicked instead of linking to the .jpg in wp-content/uploads. Now, I have some questions regarding whether this is a good practice in terms of SEO and if the fact of having duplicate images or this structure is going to hurt my SEO any way . Although both images are in different urls they ultimately link to each other this way : Blog Image --> Gallery Image url --> wp-content/uploads/file.jpg Is there a better approach for this ? Thanks
Intermediate & Advanced SEO | | ricfranco

0
Looking for help with my website

Hi does any one know of a good seo company that will get results, i.e., fix site issues and get the site improving in the serps.
Intermediate & Advanced SEO | | Taiger

0
Changing my pages URL name - HELP NEEDED FAST

Hello, I need to change the URL name for a few pages on my site. The site was launched just recently, so it has no obvious ranking and traffic. My question is, what is the best practice for changing/deleting the page name? after deleting the page, should I go to Google webmaster tool and use URL Removal and remove the old page? I know that I have to also create a new XML sitemap file, but not sure about the old pages in google search result Thanks!
Intermediate & Advanced SEO | | mdmoz

0
Help with 50 Plus Penalty, No Manual Action Reported

We have a client who had been ranking in Google’s top ten organic results for 2 of his major keywords last year. Currently Bing and Yahoo ranks his site #1 for both of these terms; the ranking pages that are appearing were specifically targeted with these words. As of now, the client appears to have a 50 plus penalty for these two keywords. Appearing #76 for one term and # 60 for the other. We were thinking of submitting a reconsideration request through Google Webmaster tools, but discovered that you aren’t allowed to do that unless a Manual Action has appeared, which in this case has not. The only problem we’ve had with the site from an SEO standpoint is that we recently discovered a website that had copied some of the product descriptions verbatim. The client contacted the site owner who took it down immediately (about a month ago), but we still have not seen any improvement in rankings for these keywords. Does anyone have any ideas on how to communicate this Google and get the suspected penalty lifted if a reconsideration request is apparently not available?
Intermediate & Advanced SEO | | roundabout

0
Was my site hit by Panda or Penguin? Looking for diagnosis help

My URL is: www.westlakedermatology.com Hello Mozers, I'm looking for some help or guidance as to why my site fell off the "rankings cliff" on 9/5. In the forums I hear a lot of others with a similar issue, and some speculation it is due to a Panda refresh. However, looking at our site we have unique content with each page having over 300-400 words (so it's not light or duplicate content). We get a lot of leads that verbally tell us our content helped answer some of their questions so I'm pretty confident its good for users. Can anyone see an issue with the content on our site? In terms of Penguin, I think our backlink profile is clean, our physicians do take part in providing content to various high quality and relevant websites/blogs. But we do not buy links or do anything in violation of Google's guidelines. In terms of brand, we are the biggest dermatology and plastic surgery group in the Austin area. So any brand implications to search should be on our side. Just looking for some sort of guidance or help, any suggestions would be great! Thanks,
Adam Paddock
Intermediate & Advanced SEO | | iderma

0
Whole site blocked by robots in webmaster tools

My URL is: www.wheretobuybeauty.com.auThis new site has been re-crawled over last 2 weeks, and in webmaster tools index status the following is displayed:Indexed 50,000 pagesblocked by robots 69,000Search query 'site:wheretobuybeauty.com.au' returns 55,000 pagesHowever, all pages in the site do appear to be blocked and over the 2 weeks, the google search query site traffic declined from significant to zero (proving this is in fact the case ).This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster toolsrobots.txt file existed but did not have any entries to allow or disallow URLs - today I have removed robots.txt file completely URL re-direction within Linux .htaccess file - there are many rows within this complex set of re-directions. Developer has double checked this file and found that it is valid.I have read everything that google and other sources have on this topic and this does not help. Also checked webmaster crawl errors, crawl stats, malware and there is no problem there related to this issue.Is this a duplicate content issue - this is a price comparison site where approx half the products have duplicate product descriptions - duplicated because they are obtained from the suppliers through an XML data file. The suppliers have the descriptions from the files in their own sites.Help!!
Intermediate & Advanced SEO | | rrogers

0
Is my landing page "over-optimized"? Please help

Hello out there My website www.painterdublin.com and www.tilers-dublin.com were heavily hit by google panda update on 27.9.2012 and EMD update few days after. I lost about 70% of the traffic mainly from combination of the keywords from my domain name (painter dublin and tilers dublin) and never managed to recover from it. I am wondering if I should also concentrate on rewriting the content of both home landing pages in the terms of "KEYWORD DENSITY". Do you think my content is "OVER OPTIMIZED" for my main keywords? (painter dublin, tilers-dublin). What is the correct use? Is there any tool to guide me? I am aware I am using those terms quite often. I don't want to start deleting those terms before I know the right way to do it. Is there anybody willing to have a look at my sites and give me advice please? kind regards Jaro
Intermediate & Advanced SEO | | jarik

0
Can you use more than one meta robots tag per page?

If you want to add both "noindex, follow" and "noopd" should you add two meta robots tags or is there a way to combine both into one?
Intermediate & Advanced SEO | | nicole.healthline

0