SEOmoz crawler not crawling my site
-
We set up a new campaign in SEOmoz on Friday. It is my understanding that the preliminary crawl can cover up to 250 and this has been our experience in the past. However, the preliminary crawl only went through 2 pages. This is a larger eCommerce site with many pages. Any ideas why more pages weren't crawled?
We set up the campaign to track at the root domain level.
-
Thanks, Ruth. I see what you're saying about the crawl now. I'll go ahead and reach out the help team. Hopefully we can figure it out!
-
So that's really what I meant was "as long as you're seeing the full crawl," was that if your next crawl found the whole site it would probably be fine. It sounds like that's not the case, so I think the Help Team is your best bet - they're far more able to diagnose campaign-specific issues than I am
-
Thanks for the reply Ruth!
I was a little unclear as to part of your response. What were you referring to when you said "as long as you continue seeing the full crawl, you should be good to go"?
Where should I be seeing the full crawl? When I look at the Crawl Diagnostic Summary, it notes that the last crawl was on May 10, 2013 (the day we set up the campaign) and the next crawl is to start May 18, 2013. I received email notification that SEOmoz was successfully recrawled it May 11, 2013. Similar to the May 10 crawl, the May 11 crawl only went through 2 pages.
I also checked up on our robots.txt and tags. There isn't blocking anything the crawler on that front.
-
Hmm! Well, it could be a couple of things.
If your full crawl ends up crawling the whole site, it's possible that the preliminary crawl timed out on a request to your site. As long as you continue seeing the full crawl, you should be good to go.
If your next crawl continues showing just a few pages, I would check to make sure you're not blocking robots with either the meta robots tag, a robots.txt file, or nofollowed links. If you don't uncover anything out of the ordinary there, it could be an account- or campaign-specific error on our end; please email help@seomoz.org and our super-awesome Help Team can help get that fixed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl 4xx Errors?
Hello! When I check our website's critical crawler issues with Moz Site Crawler, I'm seeing over 1000 pages with a 4xx error. All of the pages that are showing to have a 4xx error appear to be the brand and product pages we have on our website, but with /URL at the end of each permalink. For example, we have a page on our site for a brand called Davinci. The URL is https://kannakart.com/davinci/. In the site crawler, I'm seeing the 4xx for this URL: https://kannakart.com/davinci/URL. Could this be a plugin on our site that is generating these URLs? If they're going to be an issue, I'd like to remove them. However, I'm not sure exactly where to begin. Thanks in advance for the help, -Andrew
Moz Pro | | mostcg0 -
404 : Errors in crawl report - all pages are listed with index.html on a WordPress site
Hi Mozers, I have recently submitted a website using moz, which has pulled up a second version of every page on the WordPress site as a 404 error with index.html at the end of the URL. e.g Live page URL - http://www.autostemtechnology.com/applications/civil-blasting/ Report page URL - http://www.autostemtechnology.com/applications/civil-blasting/index.html The permalink structure is set as /%postname%/ For some reason the report has listed every page with index.html at the end of the page URL. I have tried a number of redirects in the .htaccess file but doesn't seem to work. Any suggestions will be strongly appreciated. Thanks
Moz Pro | | AmanziDigital0 -
Big changes in site titles
So as I pour though some of the diagnostics data for over 100,000 pages of my site I see thousands of page title that "could" be changed. Could this cause some lost traffic for a while due to the big changes?
Moz Pro | | dvduval0 -
Open Site Explorer
Hi - open site explorer always showed backlinks to my site's blog for Wordpress, a couple of edu sites and others that ranked somewhat high. These are no longer being shown. However, they do show up on some other backlink check tools. Does anyone know why and if this should be a concern? Thanks. Don www.uniquegiftsanddecor.com
Moz Pro | | uniquegifts-2778790 -
Open Site explorer
I checked some of the results with OSE that ranked top on Goolge SERPS that have high Domain Authority,Page Authority 1 and content on the website's particular page is thin. How come they rank high when PA is very low?
Moz Pro | | Frost0 -
SEOMoz Tool Bar
Can google put a temporary ban on my IP using the SEOmoz Toolbar too many times? TY!
Moz Pro | | TP_Marketing0 -
Site Ranking Report
Hi guys, My site ranking report says that I've gone from being 1-20 for a variety of keywords in Google UK to not in the top 50. When I do a search myself I see that my site remains where it previously was (between 1-20). How reliable is the site ranking reporting on a weekly basis? Is it best to look at it monthly?
Moz Pro | | columbus0 -
SEOmoz keyword difficulty tool
Is anyone else having problems with this? Every search I do seems to throw up an error in the traffic fields.
Moz Pro | | neooptic0