Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt not working?

2

4

965

Log in to reply

levalencia1 last edited by
Hello

This is my robots.txt file

http://www.theprinterdepo.com/Robots.txt

However I have 8000 warnings on my dashboard like this:4

What am I missing on the file¿

Crawl Diagnostics Report

On-Page Properties

<dl>

<dt>Title</dt>

<dd>Not present/empty</dd>

<dt>Meta Description</dt>

<dd>Not present/empty</dd>

<dt>Meta Robots</dt>

<dd>Not present/empty</dd>

<dt>Meta Refresh</dt>

<dd>Not present/empty</dd>

</dl>

URL: http://www.theprinterdepo.com/catalog/product_compare/add/product/100/uenc/aHR0cDovL3d3dy50aGVwcmludGVyZGVwby5jb20vaHAtbWFpbnRlbmFjZS1raXQtZm9yLTQtbGo0LWxqNS1mb3ItZXhjaGFuZ2UtcmVmdWJpc2hlZA,,/

0 Errors
- No errors found!
1 Warning
- 302 (Temporary Redirect)
  
  Found about 5 hours ago
  
  <a class="more">Read More</a>
1 Reply Last reply
Reply Quote 0
levalencia1 last edited by

anyone in here? what should I do to fix this?
1 Reply Last reply
Reply Quote 0
levalencia1 @SlavaRybalka last edited by

No I havent tried but I will.

I use magento commerce which is a great ecommerce application
1 Reply Last reply
Reply Quote 0
SlavaRybalka last edited by

1. Have you tried it without product_compare?

2. Bad CMS?
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Good robots txt for magento

Dear Communtiy, I am trying to improve the SEO ratings for my website www.rijwielcashencarry.nl (magento). My next step will be implementing robots txt to exclude some crawling pages.
Does anybody have a good magento robots txt for me? And what need i copy exactly? Thanks everybody! Greetings, Bob
Technical SEO | | rijwielcashencarry040

0
What's wrong with this robots.txt

Hi. really struggling with the robots.txt file
this is it: User-agent: *
Disallow: /product/ #old sitemap
Disallow: /media/name.xml When testing in w3c.org everything looks good, testing is okay, but when uploading it to the server, Google webmaster tools gives 3 errors. Checked it with my collegue we both don't know what's wrong. Can someone take a look at this and give me the solution.
Thanx in advance! Leonie
Technical SEO | | Leonie-Kramer

1
Staging & Development areas should be not indexable (i.e. no followed/no index in meta robots etc)

Hi I take it if theres a staging or development area on a subdomain for a site, who's content is hence usually duplicate then this should not be indexable i.e. (no-indexed & nofollowed in metarobots) ? In order to prevent dupe content probs as well as non project related people seeing work in progress or finding accidentally in search engine listings ? Also if theres no such info in meta robots is there any other way it may have been made non-indexable, or at least dupe content prob removed by canonicalising the page to the equivalent page on the live site ? In the case in question i am finding it listed in serps when i search for the staging/dev area url, so i presume this needs urgent attention ? Cheers Dan
Technical SEO | | Dan-Lawrence

0
Will a timed 301 redirect work for Googlebot?

Our client is changing brand names and domain names. We know we need to 301 redirect the old domain, but for marketing reasons we want people to see a short message saying that the brand has changed and that they will be redirected. Example: | | Our concern is how, or if, Googlebot will intepret the redirect. Will this accomplish our SEO objective of moving the value of the page to the new domain, or do we need to do just a plain old fashioned 301 redirect and not even let the page load? Thanks for your help.
Technical SEO | | GOODSIR

0
Disallow: /search/ in robots but soft 404s are still showing in GWT and Google search?

Hi guys, I've already added the following syntax in robots.txt to prevent search engines in crawling dynamic pages produce by my website's search feature: Disallow: /search/. But soft 404s are still showing in Google Webmaster Tools. Do I need to wait(it's been almost a week since I've added the following syntax in my robots.txt)? Thanks, JC
Technical SEO | | esiow2013

0
No improvement from Google penalty despite a lot of work

Hi Everyone, i hope someone can help. Our site www.danbro.co.uk suddenly dropped across all keyphrases and even our brand name on 12th April. We receieved a notification of unnatural link building and so the problem was obvious. Since then we have removed a number of old blog links, forum links that we have built up over a number of years. Once removed we sent a reconsideration. There are still a number of links we are trying (but sometimes to no avail) to remove. It is quite frustrating as we do have a number of high quality links on really relevant sites and we do update our content on a regular basis. Does anyone have any other solutions of what we can be doing? changing the domain name is not an option as it is the brand name. any help is much appreciated
Technical SEO | | Townpages

0
Client accidently blocked entire site with robots.txt for a week

Our client was having a design firm do some website development work for them. The work was done on a staging server that was blocked with a robots.txt to prevent duplicate content issues. Unfortunately, when the design firm made the changes live, they also moved over the robots.txt file, which blocked the good, live site from search for a full week. We saw the error (!) as soon as the latest crawl report came in. The error has been corrected, but... Does anyone have any experience with a snafu like this? Any idea how long it will take for the damage to be reversed and the site to get back in the good graces of the search engines? Are there any steps we should take in the meantime that would help to rectify the situation more quickly? Thanks for all of your help.
Technical SEO | | pixelpointpress

0
Is blocking RSS Feeds with robots.txt necessary?

Is it necessary to block an rss feed with robots.txt? It seems they are automatically not indexed (http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-our-web-search.html) And, google says here that it's important not to block RSS feeds (http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html) I'm just checking!
Technical SEO | | nicole.healthline

0