Why my site not crawl?
-
my error in dashboard:
**Moz was unable to crawl your site on Jul 23, 2020. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster
i thing edit robots.txt
how fix that?
-
Hey - Thanks for posting.
One thing to note here is that Moz doesn't read sitemaps. We do check robots.txt files for directives, but then crawl starting at the campaign seed URL, and work our way down in depth.
If your site is still not being crawled, I would suggest reaching out to help@moz.com with the campaign in question so we can take a look.
thanks!
-
Well, do you have pages that have Noindex, nofollow? If so, please check and verify. 2nd: you can leave your robots.txt file pretty much empty except for a link to your sitemap, i.e
sitemap: https:// www. somesite.xyz /sitemap.xml
There's no need to block bots really unless specific pages need to be blocked. it wont stop bad crawlers either putting that into your robots.
-
me added in robots.txt below code:
User-agent: rogerbot
Disallow:
but not fixed and error in my dashboard
-
You could just put the sitemap location in your robots.txt, and call it a day. Blocking robots is'nt really a good thing to be honest unless you really have to.
-
thanks
with below code ? :
User-agent: rogerbot
Disallow:
-
Your blocking a crawler to access your page. Clear it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site's DA and Pa Not Show Accurate By MOZ Extension.
my site has some minor problem which about DA and PA that DA and PA do not show accurate some time my site is yts subtitle, please tell me something about it.
Feature Requests | | darkwebvampire0 -
Can Moz add an alert to email us when a competitor's site gets a new backlink?
This would be a very useful feature, and other sites are doing this, including Ahrefs.
Feature Requests | | rabbit5190 -
How to solve Moz crawl error?
I want to crawl one of my eCommerce website. In my google webmaster account, there is not crawl error. But when I want to check crawl by Moz it shows error. I have made this ecommerce site with prestashop. Google still rank up my loacal keywords. Can anybody suggest me by checking Moz crawler error by checking my online shop.
Feature Requests | | Shamimgypsy0 -
Migrate campaign from when site changes from http:// to https://
We moved from our site from http:// to https://, redirecting all traffic to the new https connection. However, in our campaign all scores have dropped (only 1 page crawled). Is there something we can do to migratie the data, or update the existing campaign to reflect this minor change to our existing site?
Feature Requests | | BentoPres0 -
Does Moz Pro provide a back link suggestion area for a particular site?
Is there a tool with Moz Pro that offers link suggestions for specific sites?
Feature Requests | | DenisZilberberg0 -
Can I embed a Moz report in my own admin site?
I'd like to be able to pull one of the reports that I have on my campaign dashboard on another site (preferably via iframe). Is this possible?
Feature Requests | | m4change0 -
Crawl diagnostic errors due to query string
I'm seeing a large amount of duplicate page titles, duplicate content, missing meta descriptions, etc. in my Crawl Diagnostics Report due to URLs' query strings. These pages already have canonical tags, but I know canonical tags aren't considered in MOZ's crawl diagnostic reports and therefore won't reduce the number of reported errors. Is there any way to configure MOZ to not consider query string variants as unique URLs? It's difficult to find a legitimate error among hundreds of these non-errors.
Feature Requests | | jmorehouse0