605 : Page banned by robots.txt
-
Hello everyone,
I need experts help here, Please suggest, I am receiving crawl errors for my site that is , X-Robots-Tag: header, or tag.
my robots.txt file is:
User-agent: *
Disallow:
-
Hey there! I just followed up on the message you sent into our help team, but I wanted to also post the answer here for reference.
It looks like the robots.txt file may have recently been changed for the site because I created a new campaign for the subdomain and I am not getting that same error. You should no longer see this error on your next campaign update or you could create a new campaign and you would no longer see the error there.
I did notice that you ran a number of crawl tests on the site since the campaign update, but the important thing to realize is that the crawl test can be cached for up to 48 hours. (I removed the crawls in this version of the screenshot for privacy.) We also cache the crawl tests from campaign crawls, so it looks like the first crawl test you ran on the 29th was cached from your campaign crawl and the two subsequent crawl tests were cached from that first crawl test.
Again, I wanted to note that it looks like there are only links to about 2 other pages (terms and privacy) that are on the specific subdomain you are tracking, so we aren't able to crawl beyond those pages. When you limit a campaign to a specific subdomain, we can only access and crawl links that are within the same subdomain.
-
I am at a lost, I can't find the issue. Let us know what Moz says.
-
I actually have come across a handful URLs that are NoIndex, I'll DM you a list once complete.
I can't be certain this is the root of the problem (I've never seen this error in the crawl report), but based on the error you said you're getting, I believe it's a great starting point.
-
Hi Logan Ray
thank you for detailed guide, all tools bot are working perfectly except moz's. My robots meta is index, follow and my robots.txt is disallow for none for all user agents. Still there is confusion that why moz is showing crawl error. I have now emailed to moz. Let's see what they reply. I will share that.
thank you
-
Hi,
This sounds like it's more related to the meta robots tag, not the robots.txt file.
Try this:
- Run a Screaming Frog crawl on your site
- Once complete, go to the Directives tab
- Look for 'NoIndex' in the 'Meta Robots 1' column (should be the 3rd column)
- If you see anything marked with that tag, remove them - unless of course you need them there for a reason, in which case you should also block that page in your robots.txt file
-
Are you able to provide a link to site (DM me if you don't want it posted on the forum)
-
I am receiving crawl error for moz only.
There is no error at google's search console. Also, I have tested at google's robots.txt testing tool. https://www.google.com/webmasters/tools/robots-testing-too
My robots.txt file is with no slash.
User-agent: *
Disallow: -
Hi Bhomes,
Try clearing you robots.txt of any content, a robots.txt with:
User-agent: *
Disallow:/
Is blocking everything from crawling your site. See: https://support.google.com/webmasters/answer/6062598?hl=en for testing and more details on robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On Page Grade & Page Optimization Tools not working
Hello! I have an issue with my on page grader and Page optimization. Over the past two weeks i get the attached errors. I searched the forums but after contacting our developer I am in a bit of a standstill. Could someone help on what triggers this? Would highly appreciate it. Best,
Moz Bar | | pranbg
Pran c5CSsHK c5CSsHK0 -
804 : HTTPS (SSL) error encountered when requesting page.
Hi there, I am attempting a crawl test and am getting "804 : HTTPS (SSL) error encountered when requesting page." with no further information to debug the issue. Site seems to be fine as far as SSL configuration goes. How can we get more debugging information? Thanks,
Moz Bar | | RickEH
Rick0 -
Moz keyword mention on-page counting errors
Hi. Moz is showing 18 mentions of the keyword 'street furniture' on this landing page https://www.broxap.com/street-furniture.html But I can only count 6 in total in the body copy and 13 if you include navigation links. This is the same on other pages too for that keyword. Does anyone know where it's counting these extra keywords from? I don't want to fall foul of keyword stuffing but as far as I can see we're not! Could Moz be miscalculating? Any help appreciated! Thanks Joe
Moz Bar | | iweb_agency0 -
Https address has different result that http in Page Optimization Score toll in Moz PRO
The following url
Moz Bar | | TrueluxGroup
https://www.whichledlight.com/t/gu10-led-bulbs has (100 score for keyword 'GU10 LED')
has different on page opmisation score results to
http://www.whichledlight.com/t/gu10-led-bulbs (73 score for keyword 'GU10 LED') Anyone know if we've set something up wrong?
Also, is this even something to worry about, does google treat them differently? We're using the Page Optimisation Tool in Moz Pro ** UPDATE ** It's worth mentioning we are using emberjs, so the website is a single page application.
We use prerender to render the pages for google.0 -
Moz Page Analysis Country different to Who.is?
If I analyse a domain with Moz Page Analysis tool, it says that the domain is hosted in the United States but if look up the same domain on who.is, the hosting location is Italy?
Moz Bar | | Marketing_Today0 -
Have there been recent changes to the On-Page Grader tool?
I use this page all the time to grade my primary and secondary keywords and for four years now have consistently managed to achieve A-C grades for all. Today however, I noticed I am flagging F grades for 50% of the keywords. I do make a note of the initial grades for all pages (as the majority of the keywords I grade using the tool are NOT added/tracked in MOZ 'campaigns'). To make sure I wasn't going mad I went back and graded some keywords I had achieved A-C grades for just 6 days ago. The changes are as follows: keyword 1 initially graded an A - 6 days later is still an A keyword 2 initially graded C and is now graded as an F keyword 3 initially graded C and now an F keyword 4 initially a C grade and now an F keyword 5 initially a B grade and has remained a B. Out of interest I then went and added one of the keywords that is now graded an F (but last week was a C) into the 'Add and manage Page Grades' section and graded once again - it's now grading this as a C. I then removed this keyword from my tracked keywords and graded again using the 'On-Page Grader' tool and ... it's back down to an 'F'. Please could you shed some light on what's going on? I don't want to track each keyword I grade over time but I do want the parameters for grading to remain the same if I use the 'On-Page Grader' tool - what's changed? I haven't received any alerts about any changes to the tool so if anyone could shed some light on any changes I would appreciate it.
Moz Bar | | em_h_uk0 -
Is there a way to get Page Authority values included in the Crawl Diagnostic .csv export?
Would be nice to have these values included so that you can sort by Page Authority. 4uF6efx.png
Moz Bar | | WebReputationBuilders0 -
Duplicate page content
Hi guys the feedback form my campaign suggests I have to much duplicate page content. I’ve had a look at the CSV file but it doesn’t seem to be abundantly clear as to which pages on my site have the duplicate content. Can anyone tell which columns I need to refer to on the sheet, to ascertain this information. Also if the content is only slightly different, will Google still consider it to be duplicate? I look forward to hearing from you
Moz Bar | | Hardley1110