Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved New Moz Bar not working in my domain
Hello, I have installed the new version of Moz Bar in Google Chrome.
Moz Bar | | Alberto D.
My problem is that it analyzes all domains except mine: https://elchesemueve.com
I also can't find my country (Spain) to configure the site.0 -
Is Moz more accurate than Ahrefs ?
When I check my this website https://joinpakarmy.com.pk/ on Moz it gives me DA 3
Moz Bar | | joinpakarmy234
But when I check it out on Ahrefs, it tells me there is no authority for this site.
So who is more accurate in the result?
thanks0 -
Looking for a Tool to Find Referring Pages of Specific URLs
Hello Everyone, We are Looking for a Tool to Find Referring Pages of Specific URLs. Please let me know if you know of a Moz tool or another tool for this need. Thanks.
Moz Bar | | Pushm5 -
If we put the disavow links in google, does MOZ crawl the same links?
I have put bad or spam links in disavow file, but still showing in MOZ backlinks. So, I want to know that Why is MOZ not removing the spam links from their system?
Moz Bar | | insidewebanalytics0 -
Will Removing or Disavowing Toxic Links Improve MOZ Domain Authority?
The vast majority of the 140 domains that link to our website are very low quality directories or and other toxic links. Only about 20-30 domains are not toxic (according to Link Research Tools confirmed by out manual inspection of these links). Would removing some of these links improve of MOZ Domain Rank? What if we cannot remove them, can NOZ detect a disavow file? In general would improving the ratio between good quality and poor quality links improve domain authority? Thanks,
Moz Bar | | Kingalan1
Alan2 -
Crawling Problem : No-data Of My Site
Hey, Folks!! So It's Been More than a Month.And Moz is not crawling my website http://www.trickypedia.com/.I have seen Moz Last update I thought they will crawl my site but No results Disappointing. My Sites Domain Authority and Page Authority is still 1. But In other Seo Tools, they are Perfectly Crawling My Website. What Would Be the reason? Can anyone Please Explain.
Moz Bar | | seothatworks010 -
Moz Keyword Tool Monthly Volume
Ive recently put together a Keyword List of about 100 keywords on the Moz Keyword Explorer tool. One keyword, aerial filming, stood out as very low search volume of 51 - 100. I took the same 100 keywords and passed them through the Google Keyword Planner by Google AdWords. Aerial Filming has an average search volume of 1k - 10k according to the Keyword Planner. Even though Keyword Planner gives me a range of 1k - 10k, the lowest number is still 10 times higher than what the Moz Keyword Explorer was indicating. This drastic difference of volume was consistent across all 100 keywords. All of the Monthly Volume numbers were divided by 10. Why does Moz Keyword Tool display a search volume that is 10x less than what Google Keyword Planner is suggesting?
Moz Bar | | fictionarts0 -
Should I exclude prepositions in tracked keywords of moz analytics?
I'm new to Moz. Just set up my trial campaign, and it had suggested many keywords. Many of the phrases that were suggested do not contain prepositions. For example, instead of something like "sporting good stores in Chicago" it suggested "sporting good stores Chicago" Today, I looked at the on-page optimization suggestions, which are (of course) suggesting that I remove prepositions from my page to rank well. Well, as you know, that is unnatural to the reader. But I suspect people are searching in higher volume, leaving the prepositions out. I know that if I were to search for a sporting goods store in Chicago, I would probably leave out "in." What should I do? Should I remove all the suggested keywords, and make them readable (which people are less like using in their search?) Do I go back to all my pages and try to optimize it for a keyword that is natural, but does not include a preposition (such as Chicago sporting goods stores) or should I be doing something else?
Moz Bar | | osaka731