Unsolved Moz crawler not crawling on my site
-
Hi all,
im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.
This is my robots.txt file : https://www.wearefutureheads.com/robots.txt
I'm not sure what else am I missing.. can anyone help
-
@teikh Hi, thanks for contacting us! Would you be able to write into help@moz.com so that our support team can take a closer look at your query please? Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawling only the Home of my website
Hello,
Product Support | | Azurius
I don't understand why MOZ crawl only the homepage of our webiste https://www.modelos-de-curriculum.com We add the website correctly, and we asked for crawling all the pages. But the tool find only the homepage. Why? We are testing the tool before to suscribe. But we need to be sure that the tool is working for our website. If you can please help us.0 -
Blocking in Robots.txt and the re-indexing - DA effects?
I have two good high level DA sites that target the US (.com) and UK (.co.uk). The .com ranks well but is dormant from a commercial aspect - the .co.uk is the commercial focus and gets great traffic. Issue is the .com ranks for brand in the UK - I want the .co.uk to rank for brand in the UK. I can't 301 the .com as it will be used again in the near future. I want to block the .com in Robots.txt with a view to un-block it again when I need it. I don't think the DA would be affected as the links stay and the sites live (just not indexed) so when I unblock it should be fine - HOWEVER - my query is things like organic CTR data that Google records and other factors won't contribute to its value. Has anyone ever blocked and un-blocked and whats the affects pls? All answers greatly received - cheers GB
Technical SEO | | Bush_JSM0 -
Unsolved Strange "?offset" URL found with content crawl issues
I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Moz Pro | | HannahPalamara
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?0 -
Unsolved Moz can't crawl my site
Moz is being blocked from crawling the following site - https://www.cleanchain.com. When looking at Robot.txt, the following is disallowing access but don't know whether this is preventing Moz from crawling too? User-agent: *
Moz Pro | | danhart2020
Disallow: /adeci/
Disallow: /core/
Disallow: /connectors/
Disallow: /assets/components/ Could something else be preventing the crawl?0 -
Unsolved /%25s
Hi Community, has anyone else had a 404 error reported by Moz, where the end of the domain is /%25s? The error comes from my blog home page https://kaydee.net/blog/ But when I look at the source code, I can't see anything that has a space at the end of the URL. I wonder if it is to do with the WordPress search? Thanks in advance for any insight.
Moz Pro | | kaydeeweb0 -
Why moz do not show website competitive metrics
why moz do not show website competitive metrics? meko.lv and why site meko.lv has high spam score? I dont see any reason for that??? metrics website is meko.lv NKpqhns
Product Support | | Mekounko0 -
Moz Pro Reports - Can I create Monthly client reports?
I am just trying to work out if I am able to create monthly SEO reports? I see that when setting up a custom report in my campaigns, i can get it to run daily, weekly or monthly. But i'm just double checking that if i set the reports to run monthly, its going to collect all of the data from that full month, not just grab the data from the most recent week? Thanks
Product Support | | SWD.Advertising0 -
Why Moz Analytics says that my site had a 13% traffic increase and Moz Pro says that the same site had 3% decrease on traffic?
Every Tuesday The first thing I do in the morning is check my updates on my Moz Campaigns... Today the numbers just don't make sense... How come Moz Analytics and Moz Pro are showing different numbers? Does any one have the same problem? Which one should i trust?
Product Support | | Felip30