Question about Syntax in Robots.txt
-
So if I want to block any URL from being indexed that contains a particular parameter what is the best way to put this in the robots.txt file?
Currently I have-
Disallow: /attachment_idWhere "attachment_id" is the parameter. Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do
Disallow: attachment_id or Disallow: attachment_id= but figured I would ask you guys first.
Thanks!
-
That's excellent Chris.
Use the Remove Page function as well - it might help speed things up for you.
-Andy
-
I don't know how but I completely forgot I could just pop those URL's in GWT and see if they were blocked or not and sure enough, Google says they are. I guess this is just a matter of waiting.... Thanks much!
-
I have previously looked into both of those documents and the issue remains that they don't exactly address how best to block parameters. I could do this through GWT but just am curious about the correct and preferred syntax for the robots.txt as well. I guess I could just look at sites like Amazon or other big sites to see what the common practices are. Thanks though!
-
Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do
It can take Google some time to remove pages from the index.
The best way to test if this has worked is hop into Webmaster Tools and use the Test Robots.txt function. If it has blocked the required pages, then you know it's just a case of waiting - you can also remove pages from within Webmaster Tools as well, although this isn't immediate.
-Andy
-
Hi there
Take a look at Google's resource on robots.txt, as well as Moz's. You can get all the information you need there. You can also let Google know about what URLs to exclude from it's crawls via Search Console.
Hope this helps! Good luck!
-
Im not a robots.txt expert by a long shot, but I found this, which is a little dated, which explained it to me in terms i could understand.
https://sanzon.wordpress.com/2008/04/29/advanced-usage-of-robotstxt-w-querystrings/
there is also a feature in Google Webmaster tools called URL parameters that lets you block URLs with set parameters for all sorts of reason to avoid duplicate content etc. I havn't used it myself but may be work looking into
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Content question please help
Would content behind a drop down on this site Https://www.homeleisuredirect.com/pool_tables/english_pool_tables/ you have to click the - more about English pool tables text under the video Work just as well for SEO as content on the page like this site http://www.pooltablesonline.co.uk/uk-slate-bed-pool-tables.asp
Intermediate & Advanced SEO | | BobAnderson0 -
"noindex, follow" or "robots.txt" for thin content pages
Does anyone have any testing evidence what is better to use for pages with thin content, yet important pages to keep on a website? I am referring to content shared across multiple websites (such as e-commerce, real estate etc). Imagine a website with 300 high quality pages indexed and 5,000 thin product type pages, which are pages that would not generate relevant search traffic. Question goes: Does the interlinking value achieved by "noindex, follow" outweigh the negative of Google having to crawl all those "noindex" pages? With robots.txt one has Google's crawling focus on just the important pages that are indexed and that may give ranking a boost. Any experiments with insight to this would be great. I do get the story about "make the pages unique", "get customer reviews and comments" etc....but the above question is the important question here.
Intermediate & Advanced SEO | | khi50 -
I have 2 Questions
what if we do the interlinking on the exact keywords? Is this comes under spam technique? For example - http://blog.payscout.com/automotive-merchant-services/ I interlink the exact keyword in the above URL. Can we use same image 2-3 times on the same website with different anchor tags? For example - http://packforcity.com/what-to-wear-in-new-orleans-in-january/ http://packforcity.com/what-to-wear-in-san-francisco-in-october/ Same image used on the website with different alt tag.
Intermediate & Advanced SEO | | AlexanderWhite0 -
Real Estate Site Question
I'm working on this site: www.aldodavico.com - who is a real estate agent in Miami. Any ideas/best practices for SEO for a site like this one? It's got about 500 pages. I've never deal with such a huge site before.
Intermediate & Advanced SEO | | mrodriguez14400 -
Website structure question - linking to categories?
Hi there, I have a video website (user uploaded clips) which are sorted into 75 categories. Now, these categories have their own pages and 90% of the traffic comes from the category keywords. All 75 categories are linked from the homepage (which is obvious, right?) AND from all video pages. Now, my question is: from SEO point of view, it is OK to link to categories from the video pages, too? I am in doubt here because: 1. I tend to think it is OK because I get a lot of traffic for the category keywords. 2. I tend to think that isn't OK because I get almost no traffic for the video pages. Any thoughts? Thanks.
Intermediate & Advanced SEO | | jasmin280 -
Question on starting again after being penalised for bad links
Hi, in a scenario where you have been heavily penalised for bad links but the quality of your site is good, If you put the exact same version of your penalised site on a new domain (with no redirects), would Google recognise it and penalise it again, or would that give it a completely fresh start? Any advice or experience with this would be much appreciated. Thanks.
Intermediate & Advanced SEO | | em_welsby1 -
What content should I block in wodpress with robots.txt?
I need to know if anyone has tips on creating a good robots.txt. I have read a lot of info, but I am just not clear on what I should allow and not allow on wordpress. For example there are pages and posts, then attachments, wp-admin, wp-content and so on. Does anyone have a good robots.txt guideline?
Intermediate & Advanced SEO | | ENSO0 -
What Questions Should I Be Asking?
I just read a discussion that was originally posted by Steve Ollington on May 22, 2011 where he states that many people are asking the wrong types of questions on this forum. He said that he wonders if he will see a shift from people asking questions on "how to rank" to questions dealing with "how to work out the best KPIs" (Key Performance Indicators - yes I had to google it). I was once told we learn more by asking questions about a topic than by just listening. I've also been told that sometimes the right question to ask is, "What questions should I be asking?" So here is my question, what types of questions should I be asking to be better at SEO? Perhaps these are some of them: Is it possible to be good at SEO when it is not a full-time job? It is very tempting to look for easy answers when you only have limited time. What are considered KPI's? Are they different for every industry? How do you know what is junk information vs what is truly good SEO advice? Is it just simply trial and error? It seems to me that if people find truly good SEO information, they aren't going to be sharing it so easily. It's the whole, "You get what you pay for". Maybe some of you can tell me more of the questions I should be asking.
Intermediate & Advanced SEO | | kadesmith1