Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Site crawl only shows homepage
-
Hi everyone,
A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage.
Does somebody have an idea why the other pages of the website are not showing up?
Thanks,
Jens -
Hi SEOchris,
Thanks for your answer. I checked the robots.txt file, changed the User Agent to Googlebot in Screaming Frog, but non of these gave new insights. For now, we don't have access yet to the server log files, but when we have, hopefully they will tell us more.
-
The crawlers may be blocked inside of the robots.txt file. In Screaming Frog change the User Agent to Googlebot and recrawl the site. If that doesn't work you'll have to check the server log files inside of the hosting account for the site. Those may tell you what the crawl doesn't work.
-
Hey Jens,
Thanks for reaching out to us!
Would you be able to head over to this link https://moz.com/help/contact and drop us a quick email. This will allow us to investigate further (and for that, we would need to access your Moz Pro account).
Looking forward to hearing from you!
Eli
UPDATE - The Campaign was set up for us to only crawl a specific subfolder as opposed to the whole domain. All sorted now
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Mozbar extension now showing spam score
I have using extension and login my account and set free trial of one month but in this case i can't see spam score in Moz bar so how I can solve this
Moz Bar | | BarshaEmiactech0 -
If we put the disavow links in google, does MOZ crawl the same links?
I have put bad or spam links in disavow file, but still showing in MOZ backlinks. So, I want to know that Why is MOZ not removing the spam links from their system?
Moz Bar | | insidewebanalytics0 -
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this: User-Agent: rogerbot Crawl-Delay: 5 Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop. Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
Moz Bar | | AllAboutShapewear2 -
Crawling Problem : No-data Of My Site
Hey, Folks!! So It's Been More than a Month.And Moz is not crawling my website http://www.trickypedia.com/.I have seen Moz Last update I thought they will crawl my site but No results Disappointing. My Sites Domain Authority and Page Authority is still 1. But In other Seo Tools, they are Perfectly Crawling My Website. What Would Be the reason? Can anyone Please Explain.
Moz Bar | | seothatworks010 -
Moz tool bar shows "No markup schema" in webpage despites having proper schema code in page.
We have proper markup schema installed in webpage (validated using data structure tool by google) but moz tool bar says "Schema.org not found on this page." What should be the problem ?
Moz Bar | | NortonSupportSEO1 -
I can't login to the Mozbar in Chrome. Just takes me to the homepage?
So, every time I press log in on the Mozbar, it takes me to the Moz home page which I'm already logged in to but there is no change on the status of the Mozbar? What should I do to remedy this? Thanks!
Moz Bar | | soapmed6 -
Moz Bar doesn't Show Anything
Hi, I use the MozBar chrome extension, and it has worked fine in the past. But lately it doesn't show any metrics. Am I missing an update or something? Thanks.
Moz Bar | | TMI.com0 -
Pinterest Back Links Showing For One Website But Not Other in OSE
Hello, I have verified my business websites on both of my pinterest accounts for the following websites Http://www.prickettproperties.com and Http://www.ChicFamilyTravels.com The backlinks for prickett shows up in OSE, but not for ChicFamily. I am curious as to why that is? Thanks in advance for your help
Moz Bar | | Prickett0
