Why isn't the Moz crawler getting all of my item pages?
-
I am stumped and Moz is being terrible to work with.
This site has about 40k pages
39,800 of them are item pages roughly.
Moz is only finding about 2400 of my pages. So they are missing most but not all of my item pages.
I do not know which item pages they are missing.
The fact that they are finding about 2k but not the rest leads me to believe the crawler is struggling with pagination.
The site is built on Magento 2 and uses the Amasty Layered Navigation extension.
Does anyone have any ideas?
-
Meghan from Moz pointed out I had Disallow: /*? in the robots.txt and that appears to have been the problem. It is the default Magento 2 setting, so not sure why they had that in there.
-
Hi Tyler, are you setting your pagination pages as nofollow? That might be causing the issue. Any chance you can post the URL and we can try crawling the site to see if we hit any issues with a crawlers like screaming frog?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Learn How to Use Fresh Web Explorer to Build Links. Get your Daily SEO Fix.
Hi there, If you have unique data or a particularly excellent resource on your site, this content can be a great link magnet. Watch The Moz Daily SEO Fix: How to Use Fresh Web Explorer to Build Links to learn how to use Fresh Web Explorer to set up alerts and track mentions of relevant keyword phrases, find link opportunities, and build links to your content. This post wraps up The Moz Daily SEO Fix for this week. To view more Moz tool tips and tricks, check out The Moz Daily SEO Fix playlist on YouTube. Thanks!
Moz Bar | | kellyjcoop2 -
Has using Moz got me banned from Google search?
Dear fellow Mozzers, For the last couple of weeks when ever I do a google search I have to enter a captcha code. Now I hardly use google anymore which is annoying ! Then I went onto support.google.com and saw this: https://support.google.com/websearch/answer/86640?hl=en&ref_topic=3378866 "Automated traffic includes: Sending searches from a robot, computer program, automated service, or search scraper Using software that sends searches to Google to see how a website or webpage ranks on Google" Is Moz alone doing this? Thanks in advance for your response. Ash
Moz Bar | | -Ash-0 -
Has MOZ stopped including Text to Code Ratio?
My Moz extension no longer displays text to code ratio. Anyone else encounter this or know why it's gone? I see it in videos and descriptions and I used to see it in my extension, but it is no longer there.
Moz Bar | | dan.bertone0 -
Problems with Moz tools
Hello, I am having real problems with Moz tools. I can log in to Moz analytics (crawl diagnostics) and analyse one site and I get 2 [internal] duplicate page errors Yet when I enter via pro.moz.com (crawl diagnostics) I am finding over 100 duplicate page content warnings? Any idea why this is happening?
Moz Bar | | McTaggart0 -
On Page reporting Issue
Hi I’ve recently downloaded my onpage analysis report for our site http://www.health2000.co.nz . The column that is titled “Missing data description Tag” says false, which I’m assuming means the meta description is present. However in the column titled “meta _description tag” there is no meta description present (nor is any meta description in the backend of my site. Could someone please explain why this would occur? I look forward to hearing from you Regards Pete
Moz Bar | | Hardley1110 -
On page grades - quick question
My hardwood flooring client just added carpet, luxury vinyl, ceramic tile, area rugs, design, and ceramic tile to his lineup - joy, joy. He really thought the months and months of working together and creating a bunch of solid, original content and promoting it and building links and seeing great results for Hardwood Flooring means he would instantly rank well for ALL flooring. I digress, sorry. I am trying to understand the on page reports a little better. When I grade the new carpet page on his site for the keyword 'carpet' it comes back with an A - when I grade it for 'Carpet, city name' it comes back with an F. Which should I be targeting for best results? 95% of my clients are concerned about local results and not getting business other than within 20-30 miles of their office. Any guidance in understanding on how BEST to utilize the on page reporting would be greatly appreciated! (I did watch the video and check out the guide) Thank you in advance for your time! Matthew
Moz Bar | | Mrupp440 -
Not getting foreign characters in crawl diagnostics .csv
The crawl diagnostics .csv file is showing high-ascii characters instead of the correct language (foreign language website) e.g. Vietnamese, Chinese (both kinds), etc. Is there a way to get this right?
Moz Bar | | trainSEM0