MOZ Crawl help
-
Our MOZ report says it crawled 1800 pages so it reports a lot of errors based on those pages. We don't have that many pages on our site. What is MOZ crawling? I updated the profile to make sure it crawls the filtered page section of Google Analytics.
-
Hey Jessica,
I took a look at your crawl diagnostics and there isn't anything that looks too out of the ordinary. You have 415 duplicate pages, which is over inflating your page count.
Additionally, when I exported your crawl diagnostics CSV I saw a mixture of static URLs and Dynamic URLs. This may mean that you have two coppies of each page being crawled, which could be leading to over inflated page totals. You might want to look at your CMS and make sure that it is not making 2 copies of each page.
Cheers,
Ryan Watson
Business Development Associate | Moz -
Hi Jessica,
Well that's a pretty good mix
Having a bunch of different kinds of errors like that could be due to a combination of issues, but I would think that first of all you have a duplicate content issue either because your cms is creating it through different urls for things like multiple categories, archive pages and that kind of thing or because you have a server setup issue that is allowing access to the same pages through multiple urls (like www vs non www urls). It might well be a combination of these things accounting for different errors so you are going to need to narrow down the data to find out exactly what is going on.
Download your error report in csv, open it in excel and select all and then do ctrl-windows-L combination. This will put your data into a filterable table. You can then go to the headers and filter (for example) only 404 pages or only duplicate title pages. Once you have filtered for a specific error have a look at the left hand column for which page or pages the problem is appearing on and then the far right hand column to see which page is linking to this page or pages. You will then have a better idea where the problem is and how it is being reached by the crawler and that will guide you on where to look to fix it. If you want to give a couple of example urls a few obvious things might stand out.
Hope that helps!
-
By filtered, I meant that we have a profile that filters out employees hitting the site. Sorry, shouldn't have mentioned it because it isn't relevant.
-
Hi Lynn,
Thank you for getting back to me. Here is an image of our diagnostics with the errors.
-
Hi Jessica,
What are the errors telling you? If it is a lot of duplicate content type errors it could be an indication that your site is linking to the same pages in multiple ways. So even though you think your site has xx number of pages, to the crawler (and search engines) it actually has more! Not sure what you mean in regards updating the profile to crawl filtered section of google analytics? If you care to give us an example of the errors and your site we might be able to give more details.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved 403 crawl error
Hi, Moz( Also reported by GSC)have reported 403 crawl error on some of my pages. The pages are actually working fine when loaded and no visible issue at all. My web developer told me that some times error issues are reported on a working pages and there is nothing to worry about.
Product Support | | ghrisa65
My question is, will the 403 error have bad consequences on my SEO/Page ranking etc. These are some of the pages that have been reported with 403 error but loading fine: https://www.medistaff24.co.uk/hourly-home-care-in-evesham/ https://www.medistaff24.co.uk/contact-us/0 -
Is the Moz site behind for this week
We have not gotten our updated stats for this week. Is anyone else having this problem? We usually see the updates by 6am on Tuesday mornings.
Product Support | | baseballbargains0 -
I have removed a subdomain from my main domain. We have stopped the subdomain completely. However the crawl still shows the error for that sub-domain. How to remove the same from crawl reports.
Earlier I had a forum as sub-domain and was mentioned in my main domain. However i have now discontinued the forum and have removed all the links and mention of the forum from my main domain. But the crawler still shows error for the sub-domain. How to make the crawler issues clean or delete the irrelevant crawl issues. I dont have the forum now and no links at the main site, bu still shows crawl errors for the forum which doesnt exist.
Product Support | | potterharry0 -
Why is Moz Crawl Diagnostics labelling pages as duplicate when they appear to be different?
Moz Crawl Diagnostics is flagging some pages on the Doorfit website as duplicate, yet the page content is completely different and not identical. Example. Page: http://www.doorfit.co.uk/locks-security/secondary-security Duplicate: http://www.doorfit.co.uk/seals-and-sealants?cat=279 Does anybody have any suggestions as to why this might be the case? Thanks
Product Support | | A_Q0 -
Duplicate Content Report: Duplicate URLs being crawled with "++" at the end
Hi, In our Moz report over the past few weeks I've noticed some duplicate URLs appearing like the following: Original (valid) URL: http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green Duplicate URL: http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green**++** These aren't appearing in Webmaster Tools, or in a Screaming Frog crawl of our site so I'm wondering if this is a bug with the Moz crawler? I realise that it could be resolved using a canonical reference, or performing a 301 from the duplicate to the canonical URL but I'd like to find out what's causing it and whether anyone else was experiencing the same problem. Thanks, George
Product Support | | webmethod0 -
Moz keeps logging me out
I thought this issue had been resolved a few months ago. Is anyone else experiencing this problem? Best,
Product Support | | ChristopherGlaeser
Christopher0 -
How to force MOZ crawlers to search in the right area for local SEO
We do local SEO over here and so our keywords are always followed by a city name, like so: "auto repair+Springfield". If there's ever any confusion within SERPS which "Springfield" the searcher is looking for we can depend on Google to pull query location to assume it's the nearest to them (it does, and is usually correct). And furthermore, if the office ever needs to check for rankings by hand (we're not in Springfield), we just set the query location to "Springfield" and our results match their. However, MOZ doesn't set a query location. Which makes sense, I suppose, because their crawlers are up in the Bay. What I'm saying is that oftentimes there's a discrepancy between how MOZ tells me they rank, and how I see them ranking, the business owner sees them ranking, and how the customer finds them ranking. As such, we can't send them reports through MOZ 😞 Has anyone else had this problem and found a solution? Should the keyword be followed by a zip code instead of a city name? This seems like it would solve the location query problem but it may not provide the same results as service+city name.
Product Support | | cwtaylor0 -
Moz Pro Account
How many campaigns can you have before you max out your Moz pro account?
Product Support | | alblaze200