Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Report shows Internal links as zero
We have recently updated our website (https://www.payumoney.com/) and created new pages like "https://www.payumoney.com/free-payment-gateway-India.html". we have internally linked all the pages, but when I check the crawl report from MOZ it shows internal links as 0 and these pages have the PA of 1 and not increasing. I am unable to find out the reason why is it happening?
On-Page Optimization | | payu_india0 -
Website Updates
What's considered a safe number when it comes to making on-site seo changes/updates for pages of a website (refreshing copy, changing title tags, optimizing images, etc)? 1 a day? 5 a day? We need to make these changes but we don't want Google to freak out and have it have the opposite effect on our ranking.
On-Page Optimization | | SEOhughesm0 -
Number of pages crawled in dropping from 4 to 2
The report on our campaign shows that up to 2 pages are being crawled now from 4. However, our site has more pages than this. We recently inserted code to allow crawlers. what can we do to resolve this? please assist.v
On-Page Optimization | | seoworx1230 -
I want to check which pages have been crawled
I would like to find out which pages have been crawled by seomoz on my site
On-Page Optimization | | seoworx1230 -
Why is SEOMOZ Crawl Diagnostics not in sync with Webmaster Tools
Currently, my Website, according to the Crawl Diagnostics Summary, has 401 'Duplicate Page Title Errors'. But in Google Webmaster Tools, under Óptimization on the Left hand Side Toolbar, if you look up HTML Improvements, there are only 4 'Duplicate Title Tags'. I have two questions re this: A) Do 'Duplicate Page Title Errors' and 'Duplicate Title Tags' have the same meaning' ? , and B) why are there 401 errors located by the former, and just 4 by the latter?
On-Page Optimization | | ABCPS0 -
How to update a crawl
Hi I crawled my site, it found lots of errors and warning which i have fixed, I told seomoz to recrawl, and it says nothing has changed... It defiantly has though! How do I make it see these changes?
On-Page Optimization | | chrisleonard0 -
Full Crawl, now what?
Hi, Ok, I logged in this morning to see my full crawl of over 250 pages. I must admit I was expecting an email to inform me and to perhaps outline what my next steps should be. During the last 7 days I have fixed most of the pages that had errors and also added more keywords, just like what was suggested when I first signed up. So, is there a suggested plan of action I should now take to help increase the rankings, perhaps I'm missing this from the email or something? Thanks!
On-Page Optimization | | Pulsar0 -
Did anyone Rankings drop massively last weekend ...Is this new google update ?
Hi all, My Site rankings took a battering in the past week , From what I have read, I know that google have updated their algorithm and supposedly it only affects less than 1% of queries but I was very surprised to have fallen in that list... Just wondered, am i the only site to have been hammered this past week ? or is it more of a case that maybe the new algorithm means I need to be more savvy with our SEO. Just posting a quick general consensus question ... thanks Sarah
On-Page Optimization | | SarahCollins0