Robots.txt advice
-
Hey Guys,
Have you ever seen coding like this in a robots.txt, I have never seen a noindex rule in a robots.txt file before - have you?
user-agent: AhrefsBot
User-agent: trovitBot
User-agent: Nutch
User-agent: Baiduspider
Disallow: /User-agent: *
Disallow: /WebServices/
Disallow: /*?notfound=
Disallow: /?list=
Noindex: /?*list=
Noindex: /local/
Disallow: /local/
Noindex: /handle/
Disallow: /handle/
Noindex: /Handle/
Disallow: /Handle/
Noindex: /localsites/
Disallow: /localsites/
Noindex: /search/
Disallow: /search/
Noindex: /Search/
Disallow: /Search/
Disallow: ?I have never seen a noindex rule in a robots.txt file before - have you?
Any pointers? -
Never seen this, doubt it's any useful as this isn't part of any search engines recommended statements to use. I don't think this would have any impact on what search engine robots would look at as it's not a statement in the robots.txt documentation.
-
Best I could find was-
Unlike disallowed pages, noindexed pages don’t end up in the index and therefore won’t show in search results. Combine both in robots.txt to optimise your crawl efficiency: the noindex will stop the page showing in search results, and the disallow will stop it being crawled
From-https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages blocked by robots
**yazılım sürecinde yapılan bir yanlışlıktı.** Sorunu hızlı bir şekilde nasıl çözebilirim? bana yardım et. ```[XTRjH](https://imgur.com/a/XTRjH)
Intermediate & Advanced SEO | | mihoreis0 -
Robots.txt gone wild
Hi guys, a site we manage, http://hhhhappy.com received an alert through web master tools yesterday that it can't be crawled. No changes were made to the site. Don't know a huge amount about the robots.txt configuration expect that using Yoast by default it sets it not to crawl wp admin folder and nothing else. I checked this against all other sites and the settings are the same. And yet 12 hours later after the issue Happy is still not being crawled and meta data is not showing in search results. Any ideas what may have triggered this?
Intermediate & Advanced SEO | | wearehappymedia0 -
Using Meta Header vs Robots.txt
Hey Mozzers, I am working on a site that has search-friendly parameters for their faceted navigation, however this makes it difficult to identify the parameters in a robots.txt file. I know that using the robots.txt file is highly recommended and powerful, but I am not sure how to do this when facets are using common words such as sizes. For example, a filtered url may look like www.website.com/category/brand/small.html Brand and size are both facets. Brand is a great filter, and size is very relevant for shoppers, but many products include "small" in the url, so it is tough to isolate that filter in the robots.txt. (I hope that makes sense). I am able to identify problematic pages and edit the Meta Head so I can add on any page that is causing these duplicate issues. My question is, is this a good idea? I want bots to crawl the facets, but indexing all of the facets causes duplicate issues. Thoughts?
Intermediate & Advanced SEO | | evan890 -
Massive URL blockage by robots.txt
Hello people, In May there has been a dramatic increase in blocked URLs by robots.txt, even though we don't have so many URLs or crawl errors. You can view the attachment to see how it went up. The thing is the company hasn't touched the text file since 2012. What might be causing the problem? Can this result any penalties? Can indexation be lowered because of this? ?di=1113766463681
Intermediate & Advanced SEO | | moneywise_test0 -
Robots.txt: Syntax URL to disallow
Did someone ever experience some "collateral damages" when it's about "disallowing" some URLs? Some old URLs are still present on our website and while we are "cleaning" them off the site (which takes time), I would like to to avoid their indexation through the robots.txt file. The old URLs syntax is "/brand//13" while the new ones are "/brand/samsung/13." (note that there is 2 slash on the URL after the word "brand") Do I risk to erase from the SERPs the new good URLs if I add to the robots.txt file the line "Disallow: /brand//" ? I don't think so, but thank you to everyone who will be able to help me to clear this out 🙂
Intermediate & Advanced SEO | | Kuantokusta0 -
URL Question and Advice on Site Architecture
Good morning one and all, i have a specific question pertaining to my Domain Migration Website URL structure. I have a computer repair business that I am re branding and my question at this point is centrally focused on how to best handle my URL naming structure that will best suite my needs for my the Search Engines and also my customers UX while not looking SPAMMY I am a web developer and SEO and I am building a SILO Site Architecture in WordPress using Pages (not Posts) so no discussion is need on the Permalink structure. I am attaching several Images below of Screen Shots of the new site that I have designed so that you may look at them and see the Silo Architecture Layout in action for the most part. OK, here we go. Looking at the Silo Mast Head, we can see that the following Main Menu items each represent a specific Silo Theme Silo Theme # 1 - COMPUTER REPAIR Silo Theme # 2 - VIRUS REMOVAL Silo Theme # 3 - PHONE REPAIR Silo Theme # 4 - NETWORKING Silo Theme # 5 - DATA RECOVERY My specific question is, if /computer-repair/ is a main silo theme (WP -Parent Page) and /laptop-repair/ is a (Child Page) of Computer Repair is the following example below (the actual URL string) going to 'trigger' a SPAM signal to either the user or GOOGLE or both?? URL String: http://www.pcmedicsoncall.com/computer-repair/laptop-repair/ Here's another example with the VIRUS REMOVAL SILO http://www.pcmedicsoncall.com/virus-removal/malware-removal/ Seeing how computer repair is the main silo theme that cannot be changed in the URL Structure (it can) but I wont change it seeing how COMPUTER REPAIR is the single largest keyword phrase used by individuals when they are looking for computer repair. Secondly, - LAPTOP REPAIR is also a Keyword Phrase that that has HIGH search queries that I am trying to rank for and that too (ideally) should also not changed! How do I deal with this situation? Or, am I seeing this in a overly paranoid way? I currently have the site allowing only my IP Address so I am afraid that the screen shots below is all that I can do on this in lieu of actually visiting the Site Currently, I have my URL Structure where Wilmington NC immediately follows the targeted keyword phrase for the Silo Theme like below http://www.pcmedicsoncall.com/virus-removal-wilmington-nc/malware-removal/ The example above, - including the location after the keyword phrase does look much more attractive and breaks it up so it does not read SPAMMY and it will help with SEO but yet another problem exists using the location after the keyword phrase which I explain in detail Below. On top of doing a complete re-branding Domain Change I am actually going to be relocating myself and my business to Charlotte, NC at the end of the summer so I have serious doubts if using Wilmington NC within the URL structure would be a wise idea considering that I will be relocating and an internal 301 Redirect on a Newly Migrated site 2-3 months after the initial site migration and site setup may have some negative impact and confuse Google and compound the situation thus much further despite the fact that it would immediately help me bounce back up with my rankings after the migration process. Thoughts a suggestions on both explained scenarios please? I have asked this specif question once already but obviously people do not read my very detailed and well thought out questions. This can also be viewed here>http://www.seomoz.org/q/need-very-urgent-advice-on-wedsite-migration-questions-please#reply_150847> Thank you Sincerely, Marshall Thompson SEOMOZ-PC-MEDICS-ON-CALL-1.jpg SEOMOZ-PC-MEDICS-ON-CALL1.jpg
Intermediate & Advanced SEO | | MarshallThompson310 -
Wordpresss Bug? Duplicate pages yet again, Adviced Needed
I have work up this morning with a large number of missing meta description, when I looked at th pages (knowing I have to missing metas) I have duplicates of my pages that look like this, These pages normally look like www.finalduties.co.uk/ NAME OF POST now i have urls with numbers in, seems as though all my blog posts have duplicated.. Now could this be down to a wordpress bug? I am so fed up, I had all my errors all cleared up only to wake and have pages like this, which is going to cause me a problem because my pages are now duplicating.. please help, advice needed from someone that knows wordpress? | Father and son battle over 13th century inheritance http://www.finalduties.co.uk/?p=1006 1 0 Court of Appeal rules against RSCPA http://www.finalduties.co.uk/?p=1007 1 0 Life insurance policy payout can skip probate process http://www.finalduties.co.uk/?p=1008 1 0 Government clamping down on inheritance tax avoidance http://www.finalduties.co.uk/?p=1010 |
Intermediate & Advanced SEO | | Chris__Chris0 -
Advice on further SEO
I am frustrated by a lack of progress for a major keyword I want to rank for. I have made several pages, optimized with Onpage and even a whole site but I can't seem to get my ratings up. I am hoping somone can take a look at my pages and efforts and offer me some advice... Keyword is "National Currency" One site is devoted to this keyword: NationalCurrencyValues This site is ranked 30th and is down 9... and this page on another site is devoted to the same keyword ranked 26th is: http://www.antiquebanknotes.com/National-Currency.aspx
Intermediate & Advanced SEO | | Banknotes0