My site is not being fully crawled
-
Our site has been crawled several times by RogerBot but each time only 6 pages are crawled even though we have more than 100 pages.
Do I need to submit my sitemap.xml to Moz?
-
Hey There,
Thanks for writing in. We actually don't have a way to submit an XML sitemap and we only crawl the site by following the HTML links in the source code. From what I am seeing in your crawl, we are find very few unique internal links on your site, so we are only able to find unique 6 pages using the HTML links.
Unfortunately, it looks like the majority of your site is in Javascript, which our crawler is not able to parse, so we don't see any Javascript links on your site. I'm afraid that a site that is primarily in Java would not be compatible with our crawler.
I apologize for the inconvenience this causes. Please let me know if you have any other questions.
Chiaryn
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing the Moz Crawl Date
Hello, I am wondering where I can change the date of Crawl by Moz. I would like to change this crawl period from one week to 2 or even 3 weeks for Moz to crawl my website. Hope to hear from anyone soon. Kind regards, Koen.
Getting Started | | Koenniiee1 -
5xx Crawl Issue might not be issues at all. Help
Hi, I ran a crawl test on our website and it came back with 900 5xx potential errors. When I started opening these links 1 by 1 I could see they were actually working. So i exported the full list of 900 and went to the website: https://httpstatus.io/ pasted the links by 100 and used that. They came back with status codes of 301 / 301 / 200 which i believe means they are okay. After reading it says that my programmer may need to see if we are blocking the MOZ BOT or to slow the MOZ BOT down. I guess I'm wondering if this is not done is the site actually having these 5xx errors when Google is Crawling or is it just showing 900 errors because of MOZ BOT but actually things are okay? I know the simple answer is to get the programmer to fix the MOZ BOT issue to know for sure but getting programmers to do things take a lot of time so I'm trying to get a better idea here. Thanks for your input.
Getting Started | | Cfarcher1 -
Moz only crawling one page of a campaign, please help
Today I set up a new campaign for a client, however the crawl has only found the home page and is saying that the URL is unavailable. The site is definitely live and the URL is correct. I have set up the campaign 3 times one with the full address (http://www.) one with www. and with just the domain name. All three of these have come page with one page crawled and "unavailable" above the URL. It is picking up the crawl issues on the page and showing domain authority but I don't know why it's not crawling other pages. Prior to setting up the campaign I did a site crawl and Moz found everything then, so I don't know why it isn't now. Please help. Thanks
Getting Started | | Wrapped0 -
How do i stop Moz from indexing my dev site?
Hey, I am new to MOZ, and I am seeing in my page reports that I have duplicate content, but this duplicate content is mostly my development site. dev.domain.com. I have this blocked in robots.txt for google. How do i stop MOZ including it in its reports? Cheers everyone.
Getting Started | | Tholomew1
Bart0 -
How Do I Scan My New Site & Grade My Work With The Robots Turned Off? For Pre-Inspection before I launch my Site?
I have a new site that has all the bots turned off so google can't index my site until I'm finished it. I've been working on this site for a couple months now optimizing and I was wondering if there was anyway I can run a preliminary scan on the site for my titles, URLs, Headers, Alt Tags and pretty much anything else that will grade my work and tell me if i did anything wrong? Can MOZ do this with the Bots turned off? Thanks
Getting Started | | Inframan0 -
Hi, I'm looking to find out why a google+ account that was rarely used has 10,000 views. I want to discover what sites it is linked to. I entered the page url but no joy. can anyone help?
I would like to find out where all this traffic is coming from. It is most likely from an out of date sales site etc, but it's important to find out as it could be the result of hacking etc. It appears the page is linked to another site and I would like to find out which one(s) Entering the page url is not getting results, can anyone help?
Getting Started | | cyganswenia0 -
How to locate page with the duplicate title? (Crawl Diagnostics - Duplicate Titles Warning)
I am looking through my crawl diagnostics and one of my errors states that a page has a duplicate title. My problem is that I do not know how to find the duplicate. Any advice here?
Getting Started | | bearpaw0 -
MOZ Starter Crawl not happeneing
Hi I added a new site 48hours + ago and the starter crawler has not even begun collecting data. Any help would be appreciated. cheers Isaac
Getting Started | | sodafizz0