Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why do i get multiple variations of my url with ?order=asc and ?view=list at the end of it in my crawl report?
I just did a crawl for one my clients to validate any error in the structure. Next thing I know is that the website have multiple variation of the same url with query like ?order=asc and ?view=list at the end of it. I am wondering why these url variations appears in the crawl I just did since bots aren't suppose to go further thant the ? normally. Just to show you a couple of url's of my crawl test. <colgroup><col width="484"></colgroup>
Moz Bar | | alexrbrg
| https://test.com/exemple/?per_page=9 |
| https://test.com/exemple/?per_page=15 |
| https://test.com/exemple/?per_page=30 |
| https://test.com/exemple/?orderby=popularity |
| https://test.com/exemple/?orderby=date |
| https://test.com/exemple/?orderby=price |
| https://test.com/exemple/?orderby=price-desc |
| https://test.com/exemple/?order=asc |
| https://test.com/exemple/?order=desc |
| https://test.com/exemple/?view=list | Thank you Guys0 -
Is Moz Bot Broken?
Moz Bot says I have millions of issues with duplicate content. One item says that it has over 50 duplicates! I downloaded the CSV it gave me to look into it. All of my products have many different ways to get them, so I've set up rel=canonicals for the master one. So, I'd assume Mozbot would ignore those. But the interesting thing I've found is that of these 50 duplicates, some of them (on the same CSV I downloaded) are completely different products, with completely different URL's. and completely different titles and completely different meta descriptions and completely different product descriptions. So, 3 questions: Is Moz bot broken? Does it not recognize canonicals? Why is it seeing 3 very different products as exactly the same product? I'm so confused and since I work for a national company, the only way I can give actual examples is through a private message session. I guess I'm wondering if anyone else has come across this issue? Because I'm completely baffled.
Moz Bar | | DIAMA0 -
Keyword used 2389 times in page
in the "on page grader" tool I get that the keyword appears 2389 times. I'ave searched for it in the developer console, thinking it might be due to invalid HTML - but the number it's appearance is no near to 2389. Can any one help me figure out why is this so? the page is http://naamanewman.co.il/photosGalleryID.aspx?id=34
Moz Bar | | digitalalchemy
keyword "שמלות כלה נפוחות" (hebrew) QDQW4xw.png0 -
Moz Temperature Widget
I've just tried to add the Moz temperature widget to my site (https) and it is getting blocked by Google Chrome for an incorrectly configured SSL certificate (apparently!) The widget embeds this page - https://mozcast.com/embed/yesterday/horizontal Which comes up with a security issue when visited in Chrome Do I need to wait for Moz to fix? Kind Regards Liam
Moz Bar | | ZaddleMarketing0 -
I have been unable to use the export button on any of the Moz pages.
Every time I click the export button on the upper right portion of the page, it says "preparing to export", but either nothing is exported or a blank document is downloaded. I was specifically working with the Fresh Web Explorer but have noticed this on all pages. Is anyone getting this export function to work?
Moz Bar | | Manseo0 -
Problems with Moz tools
Hello, I am having real problems with Moz tools. I can log in to Moz analytics (crawl diagnostics) and analyse one site and I get 2 [internal] duplicate page errors Yet when I enter via pro.moz.com (crawl diagnostics) I am finding over 100 duplicate page content warnings? Any idea why this is happening?
Moz Bar | | McTaggart0 -
Does Moz Analytics Allow for Goal Tracking?
We're currently tracking goals in GA, but would be interested to learn if that can be accomplished in Moz Analytics.
Moz Bar | | CGalownia0 -
On Page Grade - Moz Analytics
When I go to search overview I can see my on page optimization grades, but when I try to use the on page grade optimizer for different keywords and url's I get the error. "The URL you entered does not appear to be returning a page successfully. Please make sure that you've entered the URL of valid, working page." However, I know it is a valid url and I get the error even if I use the same url and same keyword I can see the grade for from the search overview. Also, if I try to regrade the page nothing happens and where my previous grade and analysis was located I now have nothing. Any information about why this is happening? Thanks for the help.
Moz Bar | | ZeroWing0