Mozbot Can Not Crawl Entire Domain
-
I'm trying to crawl Redken.com in Moz Analytics and the Search Diagnostics is only crawling 4 pages. The domain uses a "select your country" the first time you visit, and it seems as though the bot is not getting beyond that (aka, not clicking on "USA") and is therefore not crawling the rest of the domain. There is no country specific URL other than redken.com.
I've tried entering both "redken.com" and "www.redken.com" as the URL, but no luck.
Any tips?
-
It's caused by the way you have build your site. If you click on redken.com - you get the choice of language. If you select "USA" you're redirected with 302 to redken.com/USA - then with 302 to redken.com/?country=USA then with 302 to redken.com I guess for browsers you store this somewhere (cookie?) - however for a simple bot (like Moz - but I have the same with Screaming Frog) - you just go back where you started = redken.com which again will start the same loop.
So - only 4 url's can be crawled. The other countries are on different url's so will not be included in the crawl.
Google bot is smarter and acts more like a real browser so will crawl the site - but Mozbot can't do that.
rgds
Dirk
Update - I actually forgot one redirect - redken.com first is redirected with 302 to redken.com/international
PS The site is horribly slow as well - and the redirect chain is certainly not helping.
-
Well, I just noticed that website is in flash! I believe non of crawl bots are able to crawl flash websites.
It seems that if I try to access redken.com it redirects me to flash version (/international).
Actually, now I can't recreate that. Super weird. Is there something "special" going on with automatic redirects? Look into that.
-
Thanks for the response!
These are the pages it crawled.
<colgroup><col width="420"></colgroup>
| http://redken.com |
| http://www.redken.com/ |
| http://www.redken.com/international/ |
| http://www.redken.com/USA |
| http://www.redken.com/?country=USA |Robots.txt looks clean, nothing that should have stopped it from crawling more.
-
Hi there.
Which pages are those 4 pages? Is your robots.txt blocking it for some reason maybe?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I view Domain Authority stats for longer than the previous 12 months?
When you view the DA chart, it goes back 12 months. There is no apparent way to see any data before that. I searched the Q&A but see no other similar questions. Any insight?
Getting Started | | cptutty0 -
How can I find out what is the list of keywords I currently use in my website?
How can I find out what is the list of keywords I currently use in my website? In other words I want to know my current state of keywords
Getting Started | | Rosalia.Perez0 -
My website does not allow all crawler to crawl, Now my question is that whether i need to give permission to moz crawler if yes then whaat is moz bot name?
My website does not permit all crawler to crawl website. Whether ii need to give permission to moz bot to crawl website or not? If yes what is the moz bot name?
Getting Started | | irteam0 -
How do I interpret Duplicate Content in a Crawl Report, when it only gives me a URL? How do I know what is duplicated on that page somewhere else?
I need help interpreting the Crawl Report for Duplicate Content. It gives me the URLs of pages that have duplicate content, but how do I know what content exactly is duplicated elsewhere? And how do I figure out where it is duplicated? Also, are there Moz Analytics articles or videos teaching you how to use each component of the analytics programs? Thanks!
Getting Started | | NancyBryan0 -
New to MOZ and working with Web Mentions. Can I use operators?
Our name is HostDime but often put as Host Dime (2 words) by news sources and other sites. How do I set up my brand mention so I only get a notice when both words appear, in order, together. I don't want "That host is a dime" and such. Can I use a +Host +Dime?"Host Dime"? Do these operators work in MOZ?
Getting Started | | hostdime0 -
Can't setup new campaign
Hi everyone, I'm trying to set up a new campaign for a website which has Cloudflare installed. After I enter the campaign name and URL the loading circle comes up and spins for a while, but then it just stays on the same page. No error message is given. I can't get to the next page of the setup campaign form sequence so that I can set up this campaign. Has anyone else had this problem and is there any fix? Thanks in advance
Getting Started | | _jrmo
James0 -
MOZ Starter Crawl not happeneing
Hi I added a new site 48hours + ago and the starter crawler has not even begun collecting data. Any help would be appreciated. cheers Isaac
Getting Started | | sodafizz0 -
Link Detox or I can use Open Site Explorer for tracking down bad links?
Here's the thing. I need to find bad external links pointing to my site. Is Link Detox the only option or I can actually use Open Site Explorer for that. If OSE is an option, please give me an idea how I need to go about it. Thanks.
Getting Started | | VinceWicks0