Meta robots or robot.txt file?
-
Hi Mozzers!
For parametric URL's would you recommend meta robot or robot.txt file?
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs.And what's the real difference between the two?
Thanks again!
Kay
-
No problem at all
-Andy
-
Thanks Andy!!!
-
Hi Kay,
If you want to disallow access to a page, then add the following to the Robots.txt file:
Disallow: /quickView
Then test this in Webmaster Tools.
If you want to tell Google not to index a page, then you need to do this at the page level using Meta Robots. However, don't do both (at least not at the same time). If you disallow access to a set of pages via Robots.txt and then at a later stage you Meta Noindex, Google won't see this because of the Disallow in the Robots.txt.
It really depends what you are trying to achieve, but it sounds like the Meta Robots is the way to go for you.
-Edit... here is an interesting read for you.
-Andy
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Has anyone ever seen Google truncating the beginning of a meta description on a mobile device?
I could not find any articles or mentions of this online, and I am wondering if it has to do with the website being an "m-dot" website and not responsive. Any thoughts would be appreciated! IaZJWB2
Intermediate & Advanced SEO | | accpar0 -
72KB CSS code directly in the page header (not in external CSS file). Done for faster "above the fold" loading. Any problem with this?
To optimize for googles page speed, our developer has moved the 72KB CSS code directly in the page header (not in external CCS file). This way the above the fold loading time was reduced. But may this affect indexing of the page or have any other negative side effects on rankings? I made a quick test and google cache seems to have our full pages cached, but may it affect somehow negatively our rankings or that google indexes fewer of our pages (here we have some problems with google ignoring about 30% of our pages in our sitemap".)
Intermediate & Advanced SEO | | lcourse0 -
Software assisted meta data
I was recently contacted by an SEO firm that did a search on my site and said it had a low index. Out of 5,000+ pages only 800 keywords was ranking. They said there is much improvement for adjusting my meta data for indexing. They said there is a software that does this for you. Does anyone have any experience with this? Does this sound true what they are explaining? What is this software and how much does it cost? Any help would be greatly appreciated.
Intermediate & Advanced SEO | | nchachula0 -
Search engine blocked by robots-crawl error by moz & GWT
Hello Everyone,. For My Site I am Getting Error Code 605: Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag, Also google Webmaster Also not able to fetch my site, tajsigma.com is my site Any expert Can Help please, Thanx
Intermediate & Advanced SEO | | falguniinnovative0 -
Changing meta title dynamically e.g. with number of reviews in title. Is this a problem?
I would like to put the number of reviews in the meta title of a product page. I noticed that tripadvisor for example does the same. However I read in some places that people encountered problems with frequent title tag changes. Did you ever encounter problems by just changing numbers in title tag?
Intermediate & Advanced SEO | | lcourse
Do you think it is really a problem if just the number of the reviews changes? what about putting number of reviews in the meta description instead?0 -
Robots.txt unblock
I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account that I'm denying Googlebot access to the entire site. I uploaded clean and "Allow" robots.txt yesterday, but receive the same error. I've tried "Fetch as Googlebot" on the index and other pages, but still the error. Here is the latest: | Denied by robots.txt |
Intermediate & Advanced SEO | | Elchanan
| 11/9/11 10:56 AM | As I said, there in not blocking on the robots.txt for 24 hours. HELP!0 -
Search Engine Blocked by robots.txt for Dynamic URLs
Today, I was checking crawl diagnostics for my website. I found warning for search engine blocked by robots.txt I have added following syntax to robots.txt file for all dynamic URLs. Disallow: /*?osCsid Disallow: /*?q= Disallow: /*?dir= Disallow: /*?p= Disallow: /*?limit= Disallow: /*review-form Dynamic URLs are as follow. http://www.vistastores.com/bar-stools?dir=desc&order=position http://www.vistastores.com/bathroom-lighting?p=2 and many more... So, Why should it shows me warning for this? Does it really matter or any other solution for these kind of dynamic URLs.
Intermediate & Advanced SEO | | CommercePundit0