How to crawl the whole domain?
-
Hi,
I have a website an e-commerce website with more than 4.600 products. I expect that Seomoz scan check all url's. I don't know why this doesn't happens.
The Campaign name is Artigos para festa and should scan the whole domain festaexpress.com. But it crels only 100 pages
I even tried to create a new campaign named Festa Express - Root Domain to check if it scans but had the same problem it crawled only 199 pages.
Hope to have a solution.
Thanks,
Eduardo -
Hi Kery, thanks, I just sent to them.
Regards,
Eduardo. -
Hi Eduardo,
I'm sorry you're still having problems. At this point, it'd be best for you to send an email to help@seomoz.org and have our help team look at it for you. They'd be the ones who could give you the most advice for diagnosing this.
Keri
-
Still have the same problem. Isn't that an issue with SEOMoz?
The domain is www.festaexpress.com has no flash and is crawled by google with no issues.Regards,
Eduardo. -
Hi Eduardo.
The way crawlers work is the begin on your home page and "crawl". They look at all the links on your home page and follow each one to the next page, then the next until your whole site has been captured.
Why are only 100 pages being crawled?
Most likely either because your site is not very well linked, or because you don't have a good navigation system, or because your navigation and links are presented in a format such as flash which the crawler cannot read.
Another possibility would be if the crawler is being blocked or hindered by your robots.txt file.
-
Not sure, but you could try Microsoft's IIS tool to spider your site. It is possible that your site has issues that make it difficult to spider, hence why SEOMoz's bot isn't working. You could also try something like Xenu Link Sleuth or HTTrack.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Accurate Google Keyword Ranking for a domain
Hi Folks, SEO N00b here, in a new job so please go easy on me if i ask stupid questions. I'm looking for some clarification on Google Keyword rankings. I am looking for accurate Google Keywords rankings as when I log into Google Webmaster Tools I am given values like (6-10) for a ranking? When I used SEOMoz (Research Tools -> Keyword & Ranking -> Rank Tracker) it gives me an even less accurate range by simply saying >#50?Why is SEOMoz giving a completely useless metric? Why is its actual rank not shown? I know rankings are becoming less useful as a metric with Google moving toward personalized search but I still have reports to deliver to clients so the conversation of what metrics are going in the report is to be had another day. Right now I just want to get accurate rankings, from what I've found online Google Webmaster tools is recommended as the de-facto ranking tool but it doesn't provide accurate results so how can that be (yes I know it averages out the ranking but 6-10 is a range of values not a single value that represents an average) Dave
Moz Pro | | icanseeu0 -
What is more important, page authority or linking root domains?
I have 2 pages in my report that are identical: www.ferringway.com (more page authority......21 vs. 1) www.ferringway.com/index.php (more linking root domains...... 40 vs. 21) Which one should I delete or should I do something like a 301 redirect? Your help is greatly appreciated! Sheryl
Moz Pro | | TOMMarketingLtd.0 -
Domain Authority
Has anyone ever experienced a domain authority of 1, 0 links for web 2.0 websites like tumblr, wordpress, weebly, etc? I know it's normal to see a page authority of 1 if there are no links to a url or if opensite explorer hasn't been updated. Is this some kind of bug with opensite explorer?
Moz Pro | | theanglemedia0 -
How do I fix a duplicate content error with a top level domain?
Hi, I'm getting a duplicate content error from the SEOmoz crawler due to an issue with trailing slashes. It's showing www.milengo.com and www.milengo.com/ as having duplicate page titles. However I'm pretty sure this has been fixed in the .htaccess file since if you type in the domain with a trailing slash it automatically redirects to the domain without a trailing slash, so this shouldn't be an issue. I'm stuck here. Any ideas? Thanks. Rob
Moz Pro | | milengo0 -
Domain.com and domain.com/index.html duplicate content in reports even with rewrite on
I have a site that was recently hit by the Google penguin update and dropped a page back. When running the site through seomoz tools, I keep getting duplicate content in the reports for domain.com and domain.com/index.html, even though I have a 301 rewrite condition. When I test the site, domain.com/index.html redirects to domain.com for all directories and root. I don't understand how my index page can still get flagged as duplicate content. I also have a redirect from domain.com to www.domain.com. Is there anything else I need to do or add to my htaccess file? Appreciate any clarification on this.
Moz Pro | | anthonytjm0 -
My Campaign has been crawling for about a week now
Can anyone tell me why one of my campaigns has been stuck in crawl mode for about a full week and it is still not done?!?!
Moz Pro | | nazmiyal0 -
Change campaign domain name
I have just changed the domain name on one of my sites, can i change the domain name of the reletive campaigns and keep the data, or do i have to set up a new campaigns?
Moz Pro | | AlanMosley0 -
Does it really take 3-7 days to crawl a site?
This is what you say on your website: Please note: depending on the size and speed of your site, it may take between 3 and 7 days to complete your crawl. Are you guys kidding? This is unacceptable for almost all of my deadlines, and I suspect 99% of the SEO world... you can fly to the moon faster.
Moz Pro | | rldonnell0