Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can the Lightboxes on My Site be Crawled?
I'm trying to optimize my site, but I have lightboxes and I don't know if they are visible to the search engines. If they aren't, could you suggest something that I could do? THANK YOU so much!!!!! My site is lymphexpo.com
On-Page Optimization | | bosleypalmer0 -
I am trying to better understand solving the duplicate content issues highlighted in your recent crawl report of our site - www.thehomesites.com.
Below are some of the urls highlighted as having duplicate content -
On-Page Optimization | | urahul
http://www.thehomesites.com/zip_details/76105
http://www.thehomesites.com/zip_details/44135
http://www.thehomesites.com/zip_details/75227
http://www.thehomesites.com/zip_details/94501 These are neighborhood reports generated for 4 different zip codes. We use a standard template to create these reports. What are some of the steps we can take to avoid these pages being categorized as duplicate content?0 -
How updating a post can influence seo
I have just read this great post from moz blog: http://moz.com/blog/google-fresh-factor But I haven't found nothing about updating post date. If I edit an old great post (now ranked 2nd after several months in 1st serp position) does it be better if I also update the date of the post in the wordpress post edit page? Thank you very much and sorry for my poor english. Bye, Dario.
On-Page Optimization | | Italianseolover0 -
Massive increase in Moz crawl.
I have a subdomain which has just started to be crawled by Moz, Previously this wasn't the case. The sub-domain had 16,000+ issues. Why has Moz started to count sub-domains as part of the main domain, has Google started to do this aswell?
On-Page Optimization | | danwebman0 -
How do I get rid of crawl errors?
I recently revamped all my category pages to make them Google friendly. I did a lot of keyword research and ended up deleting categories and renaming categories. Now Google Webmasters is showing a ton of 404 errors. How can I get rid of them? Do I have to remove the URL one by one with Google Webmasters? Is it something that won't hurt my rankings and will go away in time?
On-Page Optimization | | dealblogger0 -
Does it make any difference to your SEO if the homepage infrequently updates.
Hi everyone! Working on a site and the homepage hardly ever updates. Probably 3/4 times per year. The only page on the site which does is the blog page which updates 4 times per week. Is this a bad thing? Is there anything I should attempt to do?
On-Page Optimization | | RankStealer0 -
Crawl error: duplicate title for home page
I'm seeing a duplicate title for the home page, both the static file name and the domain. like: http://domain.com
On-Page Optimization | | joshcanhelp
http://domain.com/index.cfm I know how to set this in Google Analytics but how would I make sure this isn't seen as an error? It's accounting for both a duplicate title and duplicate content. Thanks!0 -
Major update to site architecture (outline)-Is Google going to drop?
I'm working with a lawyer client who has a table-based, outdated site. Her nav links consist of a jumble of topics and static pages in one long sidebar list on the home page. I'm moving her site to Wordpress and I've recommended that she organize the site based on categories that roughly match the topics/keywords she wants to rank highest for in Google. The site will be much better organized and coded and the URLs for the new launch will be much stronger for SEO by being targeted and coded properly. So the site should rank better after, right? Right??? I know that when Google crawls the new architecture, it's not going to find the expected long sidebar list of internal nav links. It'll find better, more keyword targeted internal nav links. But will that keep the site from getting dropped off page 1? I'm speaking w/ the client tomorrow and if she's going to drop or get bounced around, I feel like I should prepare her and let her know roughly what might happen. I'm thinking based on my current understanding that I should tell her to expect to be bounced around for a few weeks, but in the end she should rank higher than before. What would you do/say?
On-Page Optimization | | bvrob0