No (seomoz) crawler report since 7th may !!
-
Hi Mozteam,
I added a site less than 200 pages in the tool "seomoz crawler", at May 7.
We are May 15 and the tool always displays "crawl in progress."
Do you have a problem about this tool?
This is embarrassing ...
Thank you for your reply.
David
France
-
Hi Joel
if I tell you that I am doing SEO for 13 years you will not believe me :-))
tiredness or inattention ...thank you Joel and other contributors for your answers, I appreciate it.
David
-
Hey David,
Thanks for reaching out. I looked at your account and don't see any crawls that started on the 7th. I do see a crawl that started on the 3rd and hasn't completed because you are blocking all user agents in your robots.txt file.
You have this directive listed:
User-agent: *
Crawl-delay: 10
Disallow: /That is going to prevent all bots from accessing your site, not just RogerBot.
You'll want to either remove that or add a directive before that for User-agent: RogerBot allowing full access. Once that's done, just wait for the crawl to refresh and it should complete just fine.
Thanks!
Joel. -
Thank you,
no result at all... since may 7..
meanwhile I used Integrity .. but it is unfortunate that the report crawler does not work.
-
you have no results at all? as perfect007 said its updated once a week, so it normally says "in progress"
But if you still get no results by tomorrow, then report to the seomoz staff, along time ago I had a crawl problem where it was only crawling about 500 pages of my site, when we had 10,000+ pages. There was a technical problem with the crawl with my site which they fixed very quickly
-
Hi
thank you but it is precisely this tool I'm talking about ...
the crawl is "in progress" since May 7 ..
David
-
SEOMoz Tool crawls a website once a week, but you can do a separate crawl of any site (up to 3000 URLs) at http://pro.seomoz.org/tools/crawl-test. They generally update crawl Thursday to Thursday along with Ranking.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEOMoz API not working for Scrapebox
I want to import SEOMoz data to list of URLs I have using scrapbox. I added in my credentials according to the API but am getting error 401 as the status of all my links. Any idea why and what I should be doing?
Moz Pro | | theLotter0 -
Is SEOmoz spider ignoring my redirect?
My client was previously serving their website from both a .co.uk and a .com domain. The DNS for each of these domains was pointing to the same place, rather than redirecting. I saw this as a potential duplicate content problem so I set the .co.uk to 301 redirect on to the .com. As a user, the 301 seems to be working correctly. However, now that I have done this, SEOmoz is picking up thousands of "inbound" links from the .co.uk domain. Essentially, every single link on the internal site, is being duplicated in my stats as an inbound link as well. It appears that the spider is ignoring the redirect. I'm not sure if it's a legitimate issue that will upset Google too, or if it's just a bug with SEOMoz's spider.
Moz Pro | | MadisonSolutions0 -
Why does SEOMoz think I have duplicate content?
The SEOmoz crawl report shows me a large amount of duplicate content sites. Our site is built on a CMS that creates the link we want it to be but also automatically creates it's own longer version of the link (e.g. http://www.federalnational.com/About/tabid/82/Default.aspx and http://www.federalnational.com/about.aspx). We set the site up so that there are automatic redirects for our site. Google Webmaster does not see these pages as duplicate pages. Why does SEOmoz consider them duplicate content? Is there a way to weed this out so that the crawl report becomes more meaningful? Thanks!
Moz Pro | | jsillay0 -
On Page Ranking Tool Giving Weird Reports
My on page ranking tool is giving two entirely different reports for my website - I get one report for my domain name and a different report for the index.html even though essentially both of those pages are...well...the same. Not sure what's going on, hoping it's not an indication of a more serious issue? I appreciate any help!
Moz Pro | | Virage0 -
Why does SEOMoz crawler ignore robots.txt?
The SEOMoz crawler ignores robots.txt It also "indexes" pages marked as noindex. That means it is filling up the reports with things that don't matter. Is there any way to stop it doing that?
Moz Pro | | loopyal0 -
About NOFOLLOW tag for SEOmoz analysis
Hi all, Another issue while trying to resolve all the duplicate content SEOmoz reports to me. May be some of you guys can help: I have a dynamic error page on our website, generated in case of error, that can happen on many urls. Of course that one should not be indexed. I added the following tag on the HEADER: name="robots" content="NOODP,NOINDEX,NOFOLLOW" /> To me this should prevent from having this page indexed, but also from having this page reported by SEOmoz analyzer as duplicate content. Any hints?
Moz Pro | | nuxeo0 -
Duplicate Content and Titles in SEOMoz reports
I've had to rename some of the pages on my site and also move them to different locations. I placed a rel="canonical" on the old page pointing to the new one. The reports on my PRO Dashboard are telling me that I have Duplicate Content and Page Title errors. Do the SEOMoz automated reports take the rel="canonical" link into consideration or do I need to remove these pages and do a 301 redirect from the old to the new page?
Moz Pro | | TRICORSystems0 -
Fixing errors from SEOmoz diagnostic survey
I just ran a report from the SEOmoz diagnostic survey and was surprised to see errors. How did I fix these errors? Thanks in advance for your help, I have been pleasantly surprised at the thoughtfulness and responsiveness of this community: Errors: 5XX (server error) Overly Dynamic URL 302 Temporary Redirect Too many on page links (how many is ideal?)
Moz Pro | | TheVolkinator0