How to crawl the whole domain?
-
Hi,
I have a website an e-commerce website with more than 4.600 products. I expect that Seomoz scan check all url's. I don't know why this doesn't happens.
The Campaign name is Artigos para festa and should scan the whole domain festaexpress.com. But it crels only 100 pages
I even tried to create a new campaign named Festa Express - Root Domain to check if it scans but had the same problem it crawled only 199 pages.
Hope to have a solution.
Thanks,
Eduardo -
Hi Kery, thanks, I just sent to them.
Regards,
Eduardo. -
Hi Eduardo,
I'm sorry you're still having problems. At this point, it'd be best for you to send an email to help@seomoz.org and have our help team look at it for you. They'd be the ones who could give you the most advice for diagnosing this.
Keri
-
Still have the same problem. Isn't that an issue with SEOMoz?
The domain is www.festaexpress.com has no flash and is crawled by google with no issues.Regards,
Eduardo. -
Hi Eduardo.
The way crawlers work is the begin on your home page and "crawl". They look at all the links on your home page and follow each one to the next page, then the next until your whole site has been captured.
Why are only 100 pages being crawled?
Most likely either because your site is not very well linked, or because you don't have a good navigation system, or because your navigation and links are presented in a format such as flash which the crawler cannot read.
Another possibility would be if the crawler is being blocked or hindered by your robots.txt file.
-
Not sure, but you could try Microsoft's IIS tool to spider your site. It is possible that your site has issues that make it difficult to spider, hence why SEOMoz's bot isn't working. You could also try something like Xenu Link Sleuth or HTTrack.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a subdomain hurt/help a domain?
My organization's website currently hosts research databases on a subdomain. We're relatively new to MOZ and when I ran a crawl of my site, it came back with 30K issues.... 98% of which are on this subdomain. I wish I could just remove it and call it a day, however, real and active researchers use it regularly. My question is: Does a subdomain hurt/help a domain? I'm seeing mixed results via Google search and browsing this forum. I'm concerned that I will not be able to fix these 30K issues as I do have access to maintain this database... and just by looking at it you can tell it was built in 2005 and hasn't been updated from an SEO perspective since. Any suggestions?
Moz Pro | | mlothamer0 -
High Page Authrority, low Domain Authority. Good?
I am seeking links from real estate content sites to promote a free course I have about apartment investing. The response has been good from the first five sites I approached, with three granting links. In putting the linking pages on Open Site Explorer though I noticed on had a PA of 40, but a DA of 28. Of course I am grateful for the link, but usually it is the DA that is higher. What does it mean when the PA is higher than the DA, and would a link from this page help me? Any help is much appreciated.
Moz Pro | | HankR0 -
Tools that crawl 2 million page sites
Our site is about 2million pages deep, 50% of which is stale content. Yes, I know - OMG #unhygienic. Even if we get approval to get rid of half of it. SEOMoz Pro Elite only crawls 20k deep - what can i do to crawl and diagnose the whole site. Are there any tools anyone can suggest. SEOMoz??
Moz Pro | | ilhaam0 -
How can i get seomoz to crawl a campaign on demand
hi how can i get seomoz to crawl a campaign on demand instead of on a weekly basis? For example i have corrected some error warnings and on page elements and would like it to re crawl the site sooner to see how the corrections have worked? thanks
Moz Pro | | Bristolweb0 -
Is it possible to exclude pages from Crawl Diagnostic?
I like the crawl diagnostic but it shows many errors due to a forum that I have. I don't care about the SEO value of this forum and would like to exclude any pages in the /forum/ directory. Is it possible to add exclusions to the crawl diagnostic tool?
Moz Pro | | wfernley2 -
My domain has a * next to it i my dashboard what does this mean?
When two of my site appears in my Dashboard they have a * before them? e.g *my-debt.co.uk Whats does this mean?
Moz Pro | | BitsCube0 -
Is The Crawl Diagnostic tool working correctly?
The Crawl Diagnostic tool shows issues and displays a graph but they don't display the page specific results/suggestion like it used to. I get the "Congratulations, there are no pages affected by this issue!" message.
Moz Pro | | -PAUL-0 -
The value of .uk.com domains
It was a long night last night (working, not playing), therefore forgive the potential stupidness of this question... Running an OSE report on a website earlier their top linking domain was *.uk.com. This is a valid TLD, and I know for sure that the website has not acquired links direct from the uk.com website but from some-other-registered-domain.uk.com. What I am asking is does the Moz Calculation and/or the pagerank calculation give more value to these links? *.uk.com has a Domain Authority of 71 and 156,497 linking root domains. If so i'm gonna go buy me a bunch of .uk.com domains..... Opinions please.
Moz Pro | | eseyo2