Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt help

Intermediate & Advanced SEO

577

IceIcebaby last edited by

Hi Moz Community,

Google is indexing some developer pages from a previous website where I currently work:

ddcblog.dev.examplewebsite.com/categories/sub-categories

Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?

As always, your expertise is greatly appreciated,

-Reed
1 Reply Last reply
Reply Quote 0
KeriMorgret @LesleyPaone last edited by

The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
1 Reply Last reply
Reply Quote 1
LesleyPaone last edited by

Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
1 Reply Last reply
Reply Quote 1
KeriMorgret last edited by

You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Robots.txt Blocking - Best Practices

Hi All, We have a web provider who's not willing to remove the wildcard line of code blocking all agents from crawling our client's site (user-agent: *, Disallow: /). They have other lines allowing certain bots to crawl the site but we're wondering if they're missing out on organic traffic by having this main blocking line. It's also a pain because we're unable to set up Moz Pro, potentially because of this first line. We've researched and haven't found a ton of best practices regarding blocking all bots, then allowing certain ones. What do you think is a best practice for these files? Thanks! User-agent: * Disallow: / User-agent: Googlebot Disallow: Crawl-delay: 5 User-agent: Yahoo-slurp Disallow: User-agent: bingbot Disallow: User-agent: rogerbot Disallow: User-agent: * Crawl-delay: 5 Disallow: /new_vehicle_detail.asp Disallow: /new_vehicle_compare.asp Disallow: /news_article.asp Disallow: /new_model_detail_print.asp Disallow: /used_bikes/ Disallow: /default.asp?page=xCompareModels Disallow: /fiche_section_detail.asp
Intermediate & Advanced SEO | | ReunionMarketing

0
Can someone help me understand why this page is ranking so well?

Hi everyone, EDIT: I'm going to link to the actual page, please remove if there are any issues with confidentiality. Here is the page: https://www.legalzoom.com/knowledge/llc/topic/advantages-and-disadvantages-overview It's ranking #2 on Google for "LLC" This page is a couple months old and is substantially heavy in content, but not much more so than all the dozens of other pages online that are competing with it. This is a highly competitive industry and this particular domain is an extremely huge player in this industry. This new page is suddenly ranking #2 for an extremely competitive head term, arguably the most important/high volume keyword being targeted by the entire site. The page is outranking the home page, as well as the service page that exactly targets the query - the one that you would think would be the ranking page for this head term. However, this new page is somewhat of a spin-off with some additional related content about the subject, some videos, resources, a lot of internal links, etc. The first word of the title tag exactly matches the head term. I did observe that almost no other pages on the site have the exact keyword as the first word of the title tag, but that couldn't be sufficient to bring it up so high in the ranks, could it? Another bizarre thing that is happening is that Google is ignoring the Title Tag in the actual HTML (which is a specific question that is accurate to the content on the page), and re-assigning a title tag that basically looks like this: "Head Term | Brand." Why would it do this on this page? Doesn't it usually prefer more descriptive title tags? There are no external links coming up on Moz or Majestic pointing to this page. It has just a couple social shares. It's not being linked to from the home page or top nav bar on the main site. Can anyone explain how this particular page would outrank the main service page targeting this keyword, as well as other highly authoritative, older pages online targeting the same keyword? Thanks for your help!
Intermediate & Advanced SEO | | FPD_NYC

1
Robots.txt - blocking JavaScript and CSS, best practice for Magento

Hi Mozzers, I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento. I'm concerned we are blocking bots from crawling essential information for page rank. My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not? You can view our robots.txt file here Thanks, Blake
Intermediate & Advanced SEO | | LeapOfBelief

0
Is dynamic pages helps in E commerce SEO?

Whats are the best way to create dynamic pages in eCommerce website having static urls? Or what are other ways to increase/create more pages in websites.
Intermediate & Advanced SEO | | Obbserv

0
Hi I know this is cheeky but you are all so helpful on here!

hi, quick question, I've made a new instillation of wordpress at sussexchef.com/dev and I'm about to start building pages, obvoisly I'm going to move it to sussexchef.com when its all looking right. when I choose my page address links/ permalinks thingy, should I use new url names that don't already exist on the old site? or should I keep the old url names so I don't get loads of 404's, but include the "dev/" in the url name? Eg the old address sussexchef.com/home should I use sussexchef.com/dev/home or sussexchef.com/home-sussex-caterers while building the development site? I'm guessing the later my help out in google searches too? But if I use Dev in the url shurly I will have to go through almost 100 pages removing the dev/ and also changing all the links too? This would be days of work!
So confused! I'd really appreciate your help here. Ben
Intermediate & Advanced SEO | | SussexChef83

0
Help!!! Am I being Attacked???

Hello, I do not believe so much in spammy links attacks and I definitely do not believe my site is worth attacking. However, I'm seeing new links pointing to my site that I have no idea where they come from. I just spotted three articles on a poor crappy article site with exact match keywords point to me. The articles are completely unique (copyscaped them) and they were posted according to the site time stamp during Oct and Nov 2012. (And they Appear in the WMT recently discovered links from more or less the same time). What to do (besides for disavowing this domain)? Thanks
Intermediate & Advanced SEO | | BeytzNet

0
SEO on page content links help

I run a website at the bottom we have scroller box which the old SEO guy used to contain all of the crap content so we can rank for keywords not on the page and put all of the links in to spread the link juice into the other inner category pages (some of these pages are only listed on our innerpages otherwise). We are trying to remove this content and add decent content above the fold with relevant long tail keywords in (it is currently decent but could do with expanding if we are removing this large chunk of text in theSEO box and some long tail keywords will be missing if we just remove it) we can add a couple of links into this new content but will struggle to list the category pages not on the left hand navigation. If we were to list all of the pages in the left hand nav would we dilute the power going to the main pages currently or would we be in the same position we are now? For example at the minute I would say the power is mainly going to the left hand nav links and then a small amount of power to the links in the SEO content if we put these into the nav will it not dilute the power to the main pages. Thank you for your time and hopefully your help.
Intermediate & Advanced SEO | | BobAnderson

0
Need help in Title tags

Hi I have word press site and it has 24 pages and the main keywords is Stairlifts there are various kind and type of stairlifts and all the 24 page has the title tag stairslifts , i have tried to make each unique but i cannot avoid using the keyword stairslift. Will that have a negative impact if my main keyword is stairlifts. My target page for stairlifts is the home page. Any suggestions please Many thanks
Intermediate & Advanced SEO | | conversiontactics

0