Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Significant difference with DA scores on Moz Chrome app VS Website Tool
I saw a big difference between DA scores on Moz Chrome app VS Website Tool. Which DA score is the correct one? I personally believe the scores on Moz's site is most accurate. Do you happen to have issues syncing the scored to your Chrome app? Chrome App VS. Moz Research Tool (https://moz.com/researchtools/ose/)
Moz Bar | | iPrice_Marketing
Moz.com: 92 / 88 Here's a screenshot of the difference: https://ipricegroup-my.sharepoint.com/:i:/p/jeremy_chew/ESy9lzUTC3lOl8o93Sx6LsUB8mYXo8LmYuUj2aa0xLXi1A?e=A6OrXs This was evident in other websites too:
Priceza (priceza.com.my😞 38 / 24
Shopback (shopback.my😞 43 / 41
Cuponation (https://www.cuponation.com.my/😞 27 / 250 -
How to find a list of pages with missing H1 tags
An external SEO/PPC agency did an audit of our site a little while back and said that over 10% of our pages were missing an H1 tag. I am trying to find a way to gather a full list of these in order for our web company to fix. I downloaded the Crawl Test report thinking it would include info on tags in there but it doesn't seem to. Is there a different tool I can use that will get me this information?
Moz Bar | | Lepra0 -
On-Page Grader fails?
I have optimized the following page for the norwegian word "Dusjkabinett": https://www.vvskupp.no/Produkter/Baderom/Dusjkabinett The On-Page grader gives the URL an F-grade. BUT as you can see it seems like the tool fails in giving a correct report. See attachment. Can anybody help me understand? pCyEYaWd3
Moz Bar | | moggiew0 -
How is the Moz keyword limit for each plan determined?
I'm sure it's explained somewhere in here but I haven't found it yet so hoping someone can help: I'm on the Pro Basic plan, which allows me 300 keywords. How is the 300 keywords counted? For example, if I've uploaded 100 keywords in the Keyword Difficulty tool, and am also Rank Tracking an additional 75 keywords for a client, is that 75 kw's, 100 kw's, or 175 kw's? Put another way, if I hypothetically do rank tracking for 300 keywords, does that mean I'm not able to enter additional keywords in the Keyword Difficulty tool without deleting some of the kw's from existing accounts? Thanks for the help!
Moz Bar | | flyntime_tx
Mike0 -
Moz Crawl Test Trying to Crawl Contact Form Submit Button Location?
Moz Crawl Test for some reason is trying to Crawl a contact form Widget Submit Location. My obvious guess is that obviously the crawl cannot submit to the required fields…..I believe this because they're only kicking back these errors on the pages I have a contact form widget on. http://crawfordspest.com/pest-control/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
Moz Bar | | Funk-Creative-Media
http://crawfordspest.com/tree-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/lawn-care/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/specialty-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404 Can you shed any insight to this? I'm a bit worried that I'll have to complete gut the contact form which was one of the major requests my client requested. Or in a worse scenario make all fields not required. It would let so much spam in. I have never seem anything like this at all. But I've learned a lot from Moz, and with major errors like 404 damage Domain Authority greatly. I've fixed 404 issues with newly acquired clients existing sites and tracked through Moz and the domain authority flies up once these errors are fixed. Along with fixing what Webmaster Tools through Google reports back. ..... Let me know if you have any expertise on this matter.0 -
On Page Grader - We have keyword in Meta Description But does not show up
The on-page grader is not counting/picking up the keyword in the meta description. We have double checked the meta description and re-uploaded the page, and checked the source code on-line. The keyword is there in meta description, but not showing up in on-page-grader. https://www.trustdeedrealestateinvesting.com/trimarkfunding/buy-first-trust-deeds.html keyword buy first trust deeds
Moz Bar | | Manifestation0 -
How Can I intreptret The Crawl Report Resulst?
Hello, I am new to Moz and I have received 2 crawl reports. The first one was ok. I made a few changes to my site plugins, and my next crawl report came up with 41 4XX errors. Basically, a lot of my posts. I went back to my plugins and saw the following plugins: 404 redirect plugin & Utlimate Tiny MCE I reactivated both. I am presuming that these must have caused the issues or maybe my site was hacked. I re ran a crawl this morning, but I don't know what the different headings mean or how to understand the report. Can anyone advise? My site is new and just started to go up the rankings...so quite disappointed with this set back. regards Chriss
Moz Bar | | chrisspell0 -
Delete On Page Reports, Modify Keywords
Hi, I tried modifying the incorrect keyword assignments for my pages but there is no pencil icon. I also tried deleting the incorrect reports and redoing them but the incorrect OP reports are not deleted and the new reports are not saved when I refresh I tried following the instructions here http://moz.com/help/guides/search-overview/on-page-optimization/manage-page-grades but they didn't work. Thanks
Moz Bar | | xvpn9020