Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help

@teikh Hi, thanks for contacting us! Would you be able to write into help@moz.com so that our support team can take a closer look at your query please? Thanks

Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help

@teikh Hi, thanks for contacting us! Would you be able to write into help@moz.com so that our support team can take a closer look at your query please? Thanks

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Unsolved Moz crawler not crawling on my site

Product Support

1315

teikh Subscriber last edited by

Hi all,

im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.

This is my robots.txt file : https://www.wearefutureheads.com/robots.txt

I'm not sure what else am I missing.. can anyone help
1 Reply Last reply
Reply Quote 0
A Former User @teikh last edited by

@teikh Hi, thanks for contacting us! Would you be able to write into help@moz.com so that our support team can take a closer look at your query please? Thanks
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unsolved Crawling error emails

crawl error

Recently we start having random error messages about crawling issue:
2024-08-30 edweek:Ok
2024-08-29 marketbrief:Err. advertise: Err, edweek:Err, topschooljobs:Ok
2024-08-23 edweek:Ok
2024-08-22 marketbrief:Err. advertise: Err, edweek:Err
2024-08-21 topschooljobs:Ok, edweek:Ok
2024-08-15 marketbrief:Ok. advertise:OK
2024-08-13 edweek:Ok
2024-08-12 marketbrief:Ok
2024-08-08 marketbrief:Ok, advertise:Ok
2024-08-03 edweek:Ok, topschooljobs:Ok
All for 2024-07 - are Ok Yesterday I set 2 more crawls for the same sites (edweek and marketbrief) and I get a morning email about original edweek site is ok (still have some problem but crawl occurs and all is fine) but for test crawl for the same site "EW Test" I just got error email.
Also I suppressed ALL email communications and frankly surprised by this email. Can you please check what is wrong with a crawler or stat collection or I don't know who produced the issues.
Product Support | | DTashjian

0
is my robot file correct

robots.txt ranking

hi, can anyone let me know if my robot file is correct. my pages and wordpress posts are being indexed but not showing in serps and wondering if my robot file is wrong https://www.in2town.co.uk/robots.txt
SEO Tactics | | blogwoman1

0
Blocking in Robots.txt and the re-indexing - DA effects?

robots.txt indexation

I have two good high level DA sites that target the US (.com) and UK (.co.uk). The .com ranks well but is dormant from a commercial aspect - the .co.uk is the commercial focus and gets great traffic. Issue is the .com ranks for brand in the UK - I want the .co.uk to rank for brand in the UK. I can't 301 the .com as it will be used again in the near future. I want to block the .com in Robots.txt with a view to un-block it again when I need it. I don't think the DA would be affected as the links stay and the sites live (just not indexed) so when I unblock it should be fine - HOWEVER - my query is things like organic CTR data that Google records and other factors won't contribute to its value. Has anyone ever blocked and un-blocked and whats the affects pls? All answers greatly received - cheers GB
Technical SEO | | Bush_JSM

0
Unsolved Moz crawler not working

crawl error

Hi Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt can the support team help ?
Moz Pro | | teikh

0
Why are only two pages of my site being crawled by MOZ?

MOZ is only crawling two pages of my site even though there are links on the homepage, and the robot.txt file allows for crawling. Our site address is www.hrparts.com.
Product Support | | hrparts

0
Can I transfer my Moz Profile points to a new profile?

I've moved jobs and that meant changing emails and handing over my Moz account to the company. Afterall, it was the company that paid for the Moz account. I was wondering if I could transfer my points/profile to a new one? Or is it a case of starting from scratch again?
Product Support | | Diamond.Dog

0
Moz Billing chat

How do I access billing chat, your page says there is one http://moz.com/help/pro/phone-support but it is difficult to find. This is urgent.
Product Support | | Thommas

0
Is there a way to export the dashboard in the new MOZ?

I would like to export the dashboard to send to a client. Is there a way to do this other than taking a screenshot?
Product Support | | webbmason

0