Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We recently updated a large guide that takes the place of the original. The original has some nice organic traffic to it and I don't want to risk losing it. Should I 301 redirect to the new version, or update all the info directly on the original page?
We don't have a lot of content that garners much non-branded organic, so this is something I don't want to risk losing. We do not have a whole lot of external links into the page either.
On-Page Optimization | | AFP_Digital1 -
My organic search dropped over 25% since March Update....Any insights?
My website is purecbdvapors.com. Any insight would be greatly appreciated! Anthony
On-Page Optimization | | Purecbdvapors0 -
Can bots crawl this homepage's content?
The website is https://ashleydouglas.com.au/ I tried using http://www.seo-browser.com/ to see if bots could see the content on the site, but the tool was unable to retrieve the page. I used mobile-friendly test and it just rendered some menu links - no content and images. I also used Fetch and Render on Search Console. The result for 'how google sees the page' and 'how a visitor sees the page' are the same and only showing the main header image. Anything below isn't shown. Does this mean that bots can't actually read all content on the page past the header image? I'm not well versed with what's going on with the code. Why are the elements below the header not rendering? Is it the theme? Plugins? Thank you.
On-Page Optimization | | nhhernandez0 -
Website Updates
What's considered a safe number when it comes to making on-site seo changes/updates for pages of a website (refreshing copy, changing title tags, optimizing images, etc)? 1 a day? 5 a day? We need to make these changes but we don't want Google to freak out and have it have the opposite effect on our ranking.
On-Page Optimization | | SEOhughesm0 -
Site update and what to do about current keyword rank
Hi im in the process of giving my site a major update as its only one page and not responsive . The the new site will have a homepage with a list of my services with a small description of each and each service will link to its own page. My one page site at the moment ranks number one in my area for a low competition keyword (moz keyword score of 13%) of course this sends me very little traffic. With my new site im also going for more keywords but these will be Moz KWD 30% and 40%. I know this will be quite hard and take some time but i'm pretty sure I know what I need to do to get there. Now my question is what do i do with my current home page (only page) that ranks for that low comp keyword?
On-Page Optimization | | juun
I dont want to lose my rank for that, so do i make a new page on my new site that is optimized for that keyword, but then I don't want to 301 my homepage to that so I guess as its such an easy word to rank for my new page would soon rank for that? What are your thoughts and advice please, thanks in advance.1 -
How much content does Google Crawl on your site?
Hi, We've had a debate around the office where some people believe that Google only crawls the first 150-200 words on a page and some people believe that they priority content that is above the fold and other people believe that all content has the same priority. Can you help us? Thanks,
On-Page Optimization | | mdorville
Matt0 -
Changing Subfolder that has been crawled before
Question: I am using a wordpress multisite and I enabled the crawl options yesterday www.abc.com/subfolder <-original but i find that www.abc.com/sub is good enough I checked the site:abc.com but I find that my pages in the /subfolder has been crawled before. Can I just change it to www.abc.com/sub or it will raise duplicate content issue?
On-Page Optimization | | joony20080 -
Pages crawled
I noticed there is a limited in the number of pages crawled on galena.org? Will this number increase over time?
On-Page Optimization | | nskislak240