Site crawl only shows homepage
-
Hi everyone,
A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage.
Does somebody have an idea why the other pages of the website are not showing up?
Thanks,
Jens -
Hi SEOchris,
Thanks for your answer. I checked the robots.txt file, changed the User Agent to Googlebot in Screaming Frog, but non of these gave new insights. For now, we don't have access yet to the server log files, but when we have, hopefully they will tell us more.
-
The crawlers may be blocked inside of the robots.txt file. In Screaming Frog change the User Agent to Googlebot and recrawl the site. If that doesn't work you'll have to check the server log files inside of the hosting account for the site. Those may tell you what the crawl doesn't work.
-
Hey Jens,
Thanks for reaching out to us!
Would you be able to head over to this link https://moz.com/help/contact and drop us a quick email. This will allow us to investigate further (and for that, we would need to access your Moz Pro account).
Looking forward to hearing from you!
Eli
UPDATE - The Campaign was set up for us to only crawl a specific subfolder as opposed to the whole domain. All sorted now
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Has anyone had to deal with Moz crawl issues on their Zendesk support site?
If so - how did you end up resolving them? For instance we have 85 "temporary redirect" errors from our Zendesk support site in our crawl error report and we don't have access to the robots.txt file through Zendesk.
Moz Bar | | zspace0 -
How do I cancel a crawl request?
I was farting around and exploring the Crawl Test tool, and accidentally sent out a crawl for a competitor's site (I wanted to see if the tool would decline to crawl without verification). I do NOT want to actually crawl that site, nor do I want the competitor to see that we requested it (for obvious reasons) - how do I cancel it?
Moz Bar | | mkbeesto0 -
Is there anyway to track on-page grader grades for a site?
As opposed to going through the process of inputting the urls and keywords every couple of weeks, is there a way of Moz to send the scores for those keywords and the domain to which the url belongs?
Moz Bar | | DigitalGuru0 -
The Moz Spam Score tells me my site has too few backlinks for such a large site. How many links per page would I need to not trigger this filter and stop appearing spammy?
Hello! One of my sites is triggering the 'too few backlinks for large site' filter. I am wondering how many backlinks I need so as not to trigger this. Many thanks for your help. Toby
Moz Bar | | T0BY0 -
How Do I Troubleshoot 804 HTTPS Crawl Error?
In my Moz crawl report I get: Crawl Error
Moz Bar | | digium
Moz encountered an error on one or more pages on your site
Error Code 804: HTTPS (SSL) Error Encountered The Moz Help Section only says: 804 HTTPS (SSL) error 804 errors result from a site with misconfigured SSL software. If Moz's crawlers cannot correctly interpret an SSL response for a home page, the crawl ends immediately. My site is publicly accessible on https - https://www.respoke.io/ And I'm not seeing any issues with my certificate. Can anyone help me out? What steps can I take to troubleshoot this error? If SSL is misconfigured, how do I configure it properly?0 -
MOZ crawl test is not reporting on all the pages on my site.
I've run the crawl test one of the sites I've taken over SEO for, however its only picking all the pages. For instance it indexes all the pages under xxxxx/us but none under xxxxx/au or xxxxx/uk The pages are being indexed as they're ranking in Google. Thanks.
Moz Bar | | ahyde0 -
Canonicals in crawling reports
The crawling reports gives info about several meta data missing, what about the lack of a canonical tag? This would be nice too... and images without alt tag (or empty).
Moz Bar | | KBC0 -
Open site explorer linking root domains
My company has been trying to increase the number of linking root domains for a specific page on our website using our PR company and press releases that are sent out and linked back to this page. This is working nicely, but the number of linking root domains is still not increasing under the "linking root domains" tab. I am noticing the correct links to this domain under "fresh web mentions" though. I know this tool can take a bit to update, but it has been quite some time and I still only see the one link.
Moz Bar | | isret_efront0