Why my site not crawl?
-
my error in dashboard:
**Moz was unable to crawl your site on Jul 23, 2020. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster
i thing edit robots.txt
how fix that?
-
Hey - Thanks for posting.
One thing to note here is that Moz doesn't read sitemaps. We do check robots.txt files for directives, but then crawl starting at the campaign seed URL, and work our way down in depth.
If your site is still not being crawled, I would suggest reaching out to help@moz.com with the campaign in question so we can take a look.
thanks!
-
Well, do you have pages that have Noindex, nofollow? If so, please check and verify. 2nd: you can leave your robots.txt file pretty much empty except for a link to your sitemap, i.e
sitemap: https:// www. somesite.xyz /sitemap.xml
There's no need to block bots really unless specific pages need to be blocked. it wont stop bad crawlers either putting that into your robots.
-
me added in robots.txt below code:
User-agent: rogerbot
Disallow:
but not fixed and error in my dashboard
-
You could just put the sitemap location in your robots.txt, and call it a day. Blocking robots is'nt really a good thing to be honest unless you really have to.
-
thanks
with below code ? :
User-agent: rogerbot
Disallow:
-
Your blocking a crawler to access your page. Clear it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can Moz add an alert to email us when a competitor's site gets a new backlink?
This would be a very useful feature, and other sites are doing this, including Ahrefs.
Feature Requests | | rabbit5190 -
Is there a way to take notes on a crawled URL?
I'm trying to figure out the best way to keep track of there different things I've done to work on a page (for example, adding a longer description, or changing h2 wording, or adding a canonical URL. Is there a way to take notes for crawled URLs? If not what do you use to accomplish this?
Feature Requests | | TouchdownTech0 -
Crawl error : 804 https (SSL) error
Hi, I have a crawl error in my report : 804 : HTTPS (SSL) error encountered when requesting page. I check all pages and database to fix wrong url http->https but this one persist. Do you have any ideas, how can i fix it ? Thanks website : https://ilovemypopotin.fr/
Feature Requests | | Sitiodev0 -
Migrate campaign from when site changes from http:// to https://
We moved from our site from http:// to https://, redirecting all traffic to the new https connection. However, in our campaign all scores have dropped (only 1 page crawled). Is there something we can do to migratie the data, or update the existing campaign to reflect this minor change to our existing site?
Feature Requests | | BentoPres0 -
Does Moz Pro provide a back link suggestion area for a particular site?
Is there a tool with Moz Pro that offers link suggestions for specific sites?
Feature Requests | | DenisZilberberg0 -
Can I embed a Moz report in my own admin site?
I'd like to be able to pull one of the reports that I have on my campaign dashboard on another site (preferably via iframe). Is this possible?
Feature Requests | | m4change0 -
Crawl diagnostic errors due to query string
I'm seeing a large amount of duplicate page titles, duplicate content, missing meta descriptions, etc. in my Crawl Diagnostics Report due to URLs' query strings. These pages already have canonical tags, but I know canonical tags aren't considered in MOZ's crawl diagnostic reports and therefore won't reduce the number of reported errors. Is there any way to configure MOZ to not consider query string variants as unique URLs? It's difficult to find a legitimate error among hundreds of these non-errors.
Feature Requests | | jmorehouse0 -
Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.
Hi All, Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue. As this is likely to affect almost everyone using Cloudflare could anyone please confirm this? In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.
Feature Requests | | N1ghteyes5