Warnings on Pages excluded from Search Engines
-
I am new to this, so my question may seem a little rookie type...
When looking at my crawl diagnostic errors there are 1604 warnings for "302 redirects". Of those 1604 warnings 1500 of them are for the same page with different product ID's on them such as: www.soccerstop.com/EMailproduct.aspx?productid=999
www.soccerstop.com/EMailproduct.aspx?productid=998In our robots.txt file we have Disallow: /emailproduct.aspx
Wouldn't that take care of this problem? If so, why is it still giving me these warning errors? It does take into account our robots.txt file when generating this report does it not?
Thanks for any help you can provide.
James -
Hi Nigel,
As I understand it, the warnings in the crawl report are simply intended to alert you to things that you may not otherwise be aware of on the site. Basically, it is telling you that Roger encountered a 302 (temporary) redirect when trying to access those URL's.
If your robots.txt is functioning the way you intended, then you should see this in the "blocked by robots.txt" section of the Crawl Notices. If this is not the case, then it would indicate that perhaps your robots.txt is not correctly configured.
If you want to know exactly how Rogerbot interacts with the files on your site when crawling, you could email the help team help [at] seomoz.org and ask.
There is a very good post from Lindsay on Robot access in the SEOmoz Blog which is definitely worth a read.
Hope that helps,
-
Try this to block pages with parameters
Disallow: /emailproduct.aspx?*
You can also add a robots nofollow, noindex on the EMailproduct.aspx page if you are trying to block it
You can do parameter handling in Google Webmaster Tools > site configutation > URL Parameters as well to tell Google how to deal with the parameters.
Also, you should always do 301 directs, not 302. Good luck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I prevent duplicate page title errors from being generated by my multiple shop pages?
Our e-commerce shop has numerous pages within the main shop page. Users navigate through the shop via typical pagination. So while there may be 6 pages of products it's all still under the main shop page. Moz keeps flagging my shop pages as having duplicate titles (ie shop page 2). But they're all the same page. Users aren't loading unique pages each time they go to the next page of products and they aren't pages I can edit. I'm not sure how to prevent this issue from popping up on my reports.
Technical SEO | | NiteSkirm0 -
Should search pages be indexed?
Hey guys, I've always believed that search pages should be no-indexed but now I'm wondering if there is an argument to index them? Appreciate any thoughts!
Technical SEO | | RebekahVP0 -
Joomla creating duplicate pages, then the duplicate page's canonical points to itself - help!
Using Joomla, every time I create an article a subsequent duplicate page is create, such as: /latest-news/218-image-stabilization-task-used-to-develop-robot-brain-interface and /component/content/article?id=218:image-stabilization-task-used-to-develop-robot-brain-interface The latter being the duplicate. This wouldn't be too much of a problem, but the canonical tag on the duplicate is pointing to itself.. creating mayhem in Moz and Webmaster tools. We have hundreds of duplicates across our website and I'm very concerned with the impact this is having on our SEO! I've tried plugins such as sh404SEF and Styleware extensions, however to no avail. Can anyone help or know of any plugins to fix the canonicals?
Technical SEO | | JamesPearce0 -
See Different Landing page for my main keyword in google search result
I have a website like http://www.bannerbuzz.com, i am promoting home page with vinyl banners keyword, but currently i can see my website's review page for vinyl banners result in google, i want to display my home page instead of review page for my keyword result in google, its frequently change, some time i can see home page for it and some time it shows review page as i attached image. i want to show my home page, so can you please help me to solve it, how can i stable my home page with main keywords. OtOXxiE.png
Technical SEO | | CommercePundit0 -
Two different page authority ranks for the same page
I happened to notice that trophycentral.com and www.trophycentral.com have two different page ranks even though there is a 301 redirect. Should I be concerned? http://trophycentral.com Page Authority: 47 Domain Authority: 42 http://www.trophycentral.com Page Authority: 51 Domain Authority: 42 Thanks!
Technical SEO | | trophycentraltrophiesandawards0 -
For large sites, best practices for pages hidden behind internal search?
If a website has 1M+ pages, with most of them being hidden behind an internal search, what's the best way to get pages included in an engine's index? Does a direct clickpath to those pages need to exist from the homepage or other major hub pages on the site? Is submitting an XML sitemap enough?
Technical SEO | | vlevit0 -
Page MozRank and MozTrust 0 for Home Page, Makes No Sense?
Hey Mozzers! I'm a bit confused by a site that is showing a 0 for home page MozRank and MozTrust, while its subdomain and root domain metrics look decent (relatively). I am posting images of the page metrics and subdomain metrics to show the disparity: http://i.imgur.com/3i0jq.png http://i.imgur.com/ydfme.png Is it normal to see this type of disparity? The home page has very little inbound links, but the big goose egg has me wondering if there is something else going on. Has anyone else experienced this? Or, does anyone have speculation as to why a home page would have a 0 MozRank while the subdomain metrics look much better? Thanks!
Technical SEO | | ClarityVentures0 -
Can I use canonical tags to merge property map pages and availability pages to their counterpart overview pages?
I have a property website, for each property are 4-5 tabs each with their own URL, these pages include the overview page which is content rich, and auxilliary pages such as maps, availability, can I use a canonical tag to merge the tabs with very little content to their corresponding overview page which is content rich? I.e. www.mywebsite.co.uk/property-1/overview This page has tabs for map, town info, availability which all have their own url i.e. www.mywebsite.co.uk/property-1/map
Technical SEO | | assertive-media
www.mywebsite.co.uk/property-1/availability
www.mywebsite.co.uk/property-1/towninfo Because these auxilary pages do not contain much content can I place a canonical tag in them pointing back to the content rich overview page at www.mywebsite.co.uk/property-1/overview?0