Robots.txt help
-
Hi Moz Community,
Google is indexing some developer pages from a previous website where I currently work:
ddcblog.dev.examplewebsite.com/categories/sub-categories
Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?
As always, your expertise is greatly appreciated,
-Reed
-
The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
-
Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
-
You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate H1 Question & Landing Page help
Hi We have 2 H1's on this page http://www.key.co.uk/en/key/heavy-duty-shelving Our webmaster has put one as display:none - but isn't this just going to look like we're keyword spamming & trying to hide it? OK now I;m looking I am seeing more wrong with this page... The width buttons at the top as h2's...& they link to facet pages? Won't this just waste crawl budget? and every product title/user guide title etc are all H2's.... I just need to put a plan together to give to our dev team on what should be updated Any tips would be great. Becky
Intermediate & Advanced SEO | | BeckyKey0 -
HELP! How do I stop scraper sites - is there any recourse?
Our site has lots of unique content and photos and it is constantly being scraped and posted on other websites. Most of these are no-name sites that pop up and exist for adwords revenue. Aside from the fact that we don't want our content being copied, this is an SEO nightmare because they often link back to us from pages that are stuffed with keywords and have very low domain authority (it's a form of negative SEO). My question is: Does anyone have experience with fighting this phenonmenon? What have you done that is effective? Does anyone have experience with a service such as http://www.dmca.com/ProtectionPro.aspx ? Does it work/is it worth it? Any input is appreciated!
Intermediate & Advanced SEO | | YairSpolter0 -
SEO direction - help needed
Hi, I've been working on a site for about 5 years. We built the traffic up to about 8k visitors/day. Although now it's dropped down over the past 2 years to about 2k visitors a day. New traffic source is mainly from SEO longtail. The whole time we have been working to improve the site. What's the best way to get some help from experts on the right direction to get traffic back up or to at least tell me the site will never work 🙂 Thanks in advance. M
Intermediate & Advanced SEO | | relientmark0 -
Will Schema help my website?
I'm doing SEO on a website, zing.co.nz, which is a soon to launch company. At the moment there is a splash sight up, which will be replaced by the real sight in a few weeks upon launch. Is it worth me putting in Schemas (for the first time) so that it is recognized as an organization? Will this effect us in the serps? Thanks for your help 🙂
Intermediate & Advanced SEO | | Startupfactory0 -
Using folder blocked by robots.txt before uploaded to indexed folder - is that OK?
I have a folder "testing" within my domain which is a folder added to the robots.txt. My web developers use that folder "testing" when we are creating new content before uploading to an indexed folder. So the content is uploaded to the "testing" folder at first (which is blocked by robots.txt) and later uploaded to an indexed folder, yet permanently keeping the content in the "testing" folder. Actually, my entire website's content is located within the "testing" - so same URL structure for all pages as indexed pages, except it starts with the "testing/" folder. Question: even though the "testing" folder will not be indexed by search engines, is there a chance search engines notice that the content is at first uploaded to the "testing" folder and therefore the indexed folder is not guaranteed to get the content credit, since search engines see the content in the "testing" folder, despite the "testing" folder being blocked by robots.txt? Would it be better that I password protecting this "testing" folder? Thx
Intermediate & Advanced SEO | | khi50 -
Need help for improving SEO App?
Please refer our guru99 https://play.google.com/store/apps/details?id=com.vector.guru99&hl=en We have tons of free materials related to SAP in the app. The problem we are facing is we do not rank for terms like SAP or SAP tutorial and discovery is an issue I have searched over the internet and found no concrete solution. Can you experts help ?
Intermediate & Advanced SEO | | Chirag7530 -
Help choosing titles
Good evening guys, I changed my titles last month, in preparation for the over optimisation penalty and the result was an instant and quite dramatic loss in traffic. I believe the reason is, the change resulted in a lot of duplicate titles. My website is similar to deviant art, but for mobile phones. So the titles include the brand of mobile phone for example. The titles were: Upload name + Brand + Content type - 3 tags - FILEID So an example would be Black Nokia wallpaper - black, abstract, grey - 12345 I changed them to Black Nokia wallpaper by artist name on domain name. But this resulted in thousands of duplicate titles and a dramatic loss in traffic. For example a user could upload 20 black wallpapers. With this in mind, I need to change my titles and fast. But I don't want to make another mistake. The one I am quite keen to try is: Black Nokia Wallpaper - Tag1, tag2 wallpapers - on domain name. So the main variable would be the name of the upload and then the 2 tags, to mix things up a little. Another option would be to throw the file ID in there somewhere? As that will always be unique. Perhaps the file ID could be in the place of the "wallpapers" after the two tags? I'd like to keep the domain name, for branding reasons. Any other suggestions are warmly welcomed. Thanks a lot.
Intermediate & Advanced SEO | | seo-wanna-bs0 -
Block all search results (dynamic) in robots.txt?
I know that google does not want to index "search result" pages for a lot of reasons (dup content, dynamic urls, blah blah). I recently optimized the entire IA of my sites to have search friendly urls, whcih includes search result pages. So, my search result pages changed from: /search?12345&productblue=true&id789 to /product/search/blue_widgets/womens/large As a result, google started indexing these pages thinking they were static (no opposition from me :)), but i started getting WMT messages saying they are finding a "high number of urls being indexed" on these sites. Should I just block them altogether, or let it work itself out?
Intermediate & Advanced SEO | | rhutchings0