Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should keywords found in a keyword tool (eg. Keyword explorer) be implemented on a page in the exact order?
Hi guys, I have a question about adding keywords found directly from a keyword tool onto a webpage. I see a keyword opportunity from Moz's Keyword Explorer, for example, "dog giardia treatment natural" and was wondering if it should be worked into the page exactly in that order? For example, I've modified the question on the page to now read "Are there any other dog Giardia treatments that are natural?" Is this good practice? Should it be exactly in the order of the keyword found in the tool? Is what I've done too spammy? Appreciate any help! Thanks
Moz Bar | | marktheshark101 -
Why do I have so many site crawl issues with my site?
Hi there, Our website is www.mormonhub.com. We have many other websites that we own as well. For some reason, our website has tens of thousands of Site Crawl Issues. We have tried looking into what's causing the problem, but can't figure it out. We do use Wordpress for our website. Any help would be greatly appreciated. Thanks! wFL8I
Moz Bar | | ThridHour0 -
What does the Bold/ Strong mean in Moz bar?
Under On-Page Elements in the Moz bar there is a Tag/ Location called Bold/ Strong. What does that mean?
Moz Bar | | TiffanyatElite0 -
Moz crawl issues: All pages keep resolving to our "cookies not enabled" page
Upon running the Moz Pro site crawler, I noticed that I received quite a bit of duplicate titles along with 302 redirects (which is our site creating a temporary 302 to our "cookies not enabled" page). How would I get around the crawler being redirected to this page? I've never ran across this issue before, despite using the crawler with sites that use the same framework as the one thats affected. Any ideas?
Moz Bar | | responsivelabs0 -
Grade my page does not work
When I want to grade this page, I get the message that page is not accesible 😞 https://www.kbc.be/site/Particulieren/Verzekeren/Uw_vrije_tijd/Uw_reisverzekering_Vakantiepolis
Moz Bar | | KBC0 -
Moz is reporting a broken link error but GWT is not
My latest Moz report is showing a 404 for: http://www.fateyes.com/how-will-googles-hummingbird-affect-your-search-ranking/”ht
Moz Bar | | gfiedel
(and showing the link this way with the characters after the last / which are not part of the page URL) Google Webmaster Tools says we have no errors. I'm wondering why there is this descrepancy and I'm wondering how I can track down where this link is originating from on our site. I've tried downloading screamingfrog and deeptrawl to no avail (java issues). I've also tried a couple of services online and installed Broken Link Checker plugin with no luck finding it. Any suggestions? Thanks in advance!0 -
Given that I am currently using a bot sniffer...How can I identify the MOZ bot in order to whitelist it?
MOZ is currently blocked from crawling my sites because I use a bot sniffer. Does anyone know how I can properly identify the MOZ bot in order to whitelist it? MOZ is using Amazon web services and thus employs thousands of dynamic IPs to crawl.
Moz Bar | | Felix_LLC0 -
Moz "Crawl Diagnostics" doesn't respect robots.txt
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like: Duplicate content Overly dynamic URLs Duplicate Page Titles The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Moz Bar | | Vitalized
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored): Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/ Many thanks for any info on this issue.0