Why my site not crawl?
-
my error in dashboard:
**Moz was unable to crawl your site on Jul 23, 2020. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster
i thing edit robots.txt
how fix that?
-
Hey - Thanks for posting.
One thing to note here is that Moz doesn't read sitemaps. We do check robots.txt files for directives, but then crawl starting at the campaign seed URL, and work our way down in depth.
If your site is still not being crawled, I would suggest reaching out to help@moz.com with the campaign in question so we can take a look.
thanks!
-
Well, do you have pages that have Noindex, nofollow? If so, please check and verify. 2nd: you can leave your robots.txt file pretty much empty except for a link to your sitemap, i.e
sitemap: https:// www. somesite.xyz /sitemap.xml
There's no need to block bots really unless specific pages need to be blocked. it wont stop bad crawlers either putting that into your robots.
-
me added in robots.txt below code:
User-agent: rogerbot
Disallow:
but not fixed and error in my dashboard
-
You could just put the sitemap location in your robots.txt, and call it a day. Blocking robots is'nt really a good thing to be honest unless you really have to.
-
thanks
with below code ? :
User-agent: rogerbot
Disallow:
-
Your blocking a crawler to access your page. Clear it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I connect my moz crawl to datastudio to list 404s in a table
Hello I currently export 404s and other 'critical' issues to a google sheet and publish them in a monthly report in data studio Is there a way to automate this process so that my monthly report is automatically populated with critical issues
Feature Requests | | Andrew-SEO0 -
Can Moz add an alert to email us when a competitor's site gets a new backlink?
This would be a very useful feature, and other sites are doing this, including Ahrefs.
Feature Requests | | rabbit5190 -
Is there a way to take notes on a crawled URL?
I'm trying to figure out the best way to keep track of there different things I've done to work on a page (for example, adding a longer description, or changing h2 wording, or adding a canonical URL. Is there a way to take notes for crawled URLs? If not what do you use to accomplish this?
Feature Requests | | TouchdownTech0 -
Crawl error : 804 https (SSL) error
Hi, I have a crawl error in my report : 804 : HTTPS (SSL) error encountered when requesting page. I check all pages and database to fix wrong url http->https but this one persist. Do you have any ideas, how can i fix it ? Thanks website : https://ilovemypopotin.fr/
Feature Requests | | Sitiodev0 -
Does moz offer a site auditor tool to imbed on your website?
similar service as http://mysiteauditor.com. just want to embed a tool on our website that allows the visitor to enter their url and have a report emailed to them.
Feature Requests | | WebMarkets1 -
Migrate campaign from when site changes from http:// to https://
We moved from our site from http:// to https://, redirecting all traffic to the new https connection. However, in our campaign all scores have dropped (only 1 page crawled). Is there something we can do to migratie the data, or update the existing campaign to reflect this minor change to our existing site?
Feature Requests | | BentoPres0 -
Can I embed a Moz report in my own admin site?
I'd like to be able to pull one of the reports that I have on my campaign dashboard on another site (preferably via iframe). Is this possible?
Feature Requests | | m4change0 -
Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.
Hi All, Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue. As this is likely to affect almost everyone using Cloudflare could anyone please confirm this? In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.
Feature Requests | | N1ghteyes5