Crawl depth seems off?
-
I'm reviewing my site crawl data and am seeing some very strange things such as:
- The homepage URL has a listed crawl depth of 2.
- Pages that are featured in the main site navigation (which is present on all pages, including homepage) are ranking at a crawl depth of 3.
What am I missing here? Shouldn't my homepage have a crawl depth of 0 or 1? Why would pages linked directly from my homepage have a crawl depth other than 1? (Single click from homepage to that page)?
Thank you!
-
Hi Samantha,
I set up a new campaign using the https:// version of the site and ran a new crawl, but I'm running into the same issue as before. Perhaps this is a bigger question of how site redirects work? I was under the impression that any large-scale redirects (such as from non-www to www or http to https across all pages) can affect crawl time/load time. Rereading your comment, it sounds like what you're saying is those redirects count as layers of crawl depth, as well. By the same token, I'm assuming any redirects (301's in particular) also add a layer of crawl depth.
So, my larger question then is: how can I maximize crawl depth if my site has been redirected from http to https? Will that "extra layer" of crawling always be there as long as the redirect is in place, or is there a way to compress/expedite how the crawl happens?
Thanks for your input on this!
-
Hi Samantha,
That makes sense, thank you. I'll set up a new campaign tracking with "https://" instead!
-
Hey there,
Sam from Moz's Help Team here!
So the thing to keep in mind when you set up a campaign at the root domain level is that we'll be starting the crawl from the http protocol (non-www). In this case - http://logic2020.com/. If you filter by crawl depth in your Site Crawl you'll see that URL with a crawl depth of 0.
It redirects to http://www.logic2020.com/ which has a crawl depth of 1. That URL then redirects again to https://www.logic2020.com/, which is listed with a crawl depth of 2 - hence why links we found on that page have a crawl depth of 3.
I hope this helps to clarify but let me know if you have any other questions!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz can't crawl our site
Moz can't crawl our site because of an error in the robots.txt, we've tried everything in the troubleshooting guide but nothing works - I believe its a server error but have no idea how to fix it pls help
Link Explorer | | SigneerHFS0 -
Moz crawling http rather than https site
Our site is secure but when I ask moz to crawl it by giving the root domain including https moz insists on crawling the non secure version. How do i force it to crawl the secure version?
Link Explorer | | media12340 -
Angular SPA & MOZ Crawl Issues
Website: https://www.exambazaar.com/ Issue: Domain Authority & Page Authority 1/100 I am using Prerender to cache/render static pages to crawl agents but MOZ is not able to crawl through my website (https://www.exambazaar.com/). Hence I think it has a domain authority of 1/100. I have been in touch with Prerender support to find a fix for the same and have also added dotbot to the list of crawler agents in addition to Prerender default list which includes rogerbot. Do you have any suggestions to fix this? List: https://github.com/prerender/prerender-node/commit/5e9044e3f5c7a3bad536d86d26666c0d868bdfff Adding dotbot to Express Server:
Link Explorer | | gparashar
prerender.crawlerUserAgents.push('dotbot');0 -
Moz Crawl Issue?
So I've been looking over my latest crawl and two of each category appears, the only difference between them is a / at the end, i.e. http://thespacecollective.com/space-clothing/ http://thespacecollective.com/space-clothing One brings in a 200 code while the other brings in a 301. Given that the 301 is in place I know Google won't see it as a duplicate, but I'm curious as to why Moz picks up on this and whether or not there is an issue that needs addressing here?
Link Explorer | | moon-boots0 -
I crawled my site, but an old crawl report still is visible
I crawled my site recently, but an old crawl report still is still all I can see
Link Explorer | | Bigjim0 -
None of the pages crawled contain an email address or links to a social profile...
I'm trying to reduce the amount of spam flags our website has using the Open Site Explorer tool. Currently, we have 2/17 flags based on: Large Site with Few Links - We found very few sites linking to this site, considering its size No Contact Info - None of the pages crawled contain an email address or links to a social profile The "few links" can be ignored, we're working on this. We don't have a visible email address on the website and we don't particularly want one. We prefer customers to fill out our online form or to call us. Moz say "none of the pages crawled contain an email address OR links to a social profile" - we do have social buttons on every page of the website, but these are official Facebook and Twitter buttons that are rendered with Javascript, so don't actually appear in the page source on load. If we replace these with actual links to our pages using Facebook and Twitter icons, will this flag be removed since Moz are saying "or links to a social profile" - making it sound optional. Thanks!
Link Explorer | | LiamMcArthur0 -
Incorrect crawl errors
A crawl of my websites has indicated that there are some 5XX server errors on my website: Error Code 608: Page not Decodable as Specified Content Encoding
Link Explorer | | LiamMcArthur
Error Code 803: Incomplete HTTP Response Received
Error Code 803: Incomplete HTTP Response Received
Error Code 608: Page not Decodable as Specified Content Encoding
Error Code 902: Network Errors Prevented Crawler from Contacting Server The five pages in question are all in fact perfectly working pages and are returning HTTP 200 codes. Is this a problem with the Moz crawler?1 -
Is there some way to tell the Moz crawler not to crawl URL's with particular dynamic tags such as "?redirect-to:http//" ?
We are encountering an issue where the crawler is finding a ton of pages from our wordpress login url that has this dynamic tag in it to kinds of different blog entries. It's madness. I can't figure out what is causing these URLs to generate to be crawled in the first place! Does this sound familiar to anyone out there, any constructive suggestions? Robots text or maybe meta robots tags that would resolve this crawl issue?
Link Explorer | | RegistrarCorp0