Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Why is my website Backlinks Lower on Moz than on SEMRush?
Hi, I'm new to Moz and I am trying to find out why my website Birmingham Roofers has 7 backlinks on Moz it has 21 on SEMRush. However, I read somewhere online in one of your blog post that it crawls in real time.
Moz Bar | Nov 2, 2023, 5:39 PM | ajegunle1 -
In MozBar Page Analysis under General Attributes Country is empty or wrong
Some of our clients sites have a top level domain for Mexico, but in the Mozbar they doesn't show any country, is this related to a known bug? https://www.edacom.mx/ https://www.dropbox.com/s/a8jgr7ztk1im15k/Captura de pantalla 2019-08-15 12.41.10.png?dl=0 https://www.cliento.mx/ https://www.dropbox.com/s/a8jgr7ztk1im15k/Captura de pantalla 2019-08-15 12.41.10.png?dl=0 In the search console: "International Targeting report" the target is Ok to Mexico.
Moz Bar | Aug 16, 2019, 5:42 AM | Andrea_Ugalde1 -
How do I see amount of traffic to a particular page on a website?
I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?
Moz Bar | May 8, 2024, 7:32 PM | jbcorcoran6 -
Moz not updating the spam score metrics
Hi Experts, I have done everything to come over the following metrics flagged as spam on my website by Moz, few months ago. But, Moz has not updated the spam score yet. ✓Low MozTrust or MozRank Score - improved from 2 to 4.5
Moz Bar | Feb 26, 2018, 8:53 AM | jamesh.rich01
✓Large Site with Few Links - My website have more than 6K backlinks
✓Small Proportion of Branded Links - My website have a good amount of branded backlinks
✓Thin Content - Every webpage on website has more than 500 words content
✓External Links in Navigation - There is no external link in navigation other than social media links
✓No Contact Info - The proper address has alreay been placed on website footer
✓Low Number of Pages Found - I am wondering if there are any standard score or number of links to reach to remove these flag?
Also, please suggest some ways that will help me improve moz spam score at faster rate. Thanks for your help in advance!"0 -
Only One Canonical URL Tag
HI, I'm an SEO novice - company owner with no money so doing it all myself with help from my web designer using wordpress. Ive just completed some seo and done the moz page scoring analysis for optimisation and gained 92% - however - there is one outstanding issue on canonical url tags - i.e. recommened fix = The canonical URL tag is intended to refer duplicate pages to a single canonical URL. To ensure the search engines properly parse the canonical source, your page should use only one version of this tag in the header. See Canonical URL Tag - the Most Important Advancement in SEO Practices Since Sitemaps Ive gone through the page code and can see I have 2 rel=canonical references - am I able to simply delete one - how do I do this if its been created by the yoast/wordpress plug-in? Many thanks in advance for any help!
Moz Bar | Mar 24, 2017, 4:11 AM | M-J-Smith0 -
How to find a list of pages with missing H1 tags
An external SEO/PPC agency did an audit of our site a little while back and said that over 10% of our pages were missing an H1 tag. I am trying to find a way to gather a full list of these in order for our web company to fix. I downloaded the Crawl Test report thinking it would include info on tags in there but it doesn't seem to. Is there a different tool I can use that will get me this information?
Moz Bar | Jul 19, 2016, 5:41 PM | Lepra0 -
Should I exclude prepositions in tracked keywords of moz analytics?
I'm new to Moz. Just set up my trial campaign, and it had suggested many keywords. Many of the phrases that were suggested do not contain prepositions. For example, instead of something like "sporting good stores in Chicago" it suggested "sporting good stores Chicago" Today, I looked at the on-page optimization suggestions, which are (of course) suggesting that I remove prepositions from my page to rank well. Well, as you know, that is unnatural to the reader. But I suspect people are searching in higher volume, leaving the prepositions out. I know that if I were to search for a sporting goods store in Chicago, I would probably leave out "in." What should I do? Should I remove all the suggested keywords, and make them readable (which people are less like using in their search?) Do I go back to all my pages and try to optimize it for a keyword that is natural, but does not include a preposition (such as Chicago sporting goods stores) or should I be doing something else?
Moz Bar | May 7, 2015, 12:40 PM | osaka731 -
Moz reporting for C-Blocking
Hey Mozers, I see Moz has a reporting tool for C-blocking and for november I had 330. Does this mean 330 Ip addresses came from the same location in the month of november?
Moz Bar | Jan 2, 2015, 1:44 PM | rpaiva1