Search engine blocked by robots-crawl error by moz & GWT
-
Hello Everyone,.
For My Site I am Getting Error Code 605: Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag, Also google Webmaster Also not able to fetch my site, tajsigma.com is my site
Any expert Can Help please,
Thanx
-
When was your last crawl date in Google Webmaster Tools/Search Console? It may be that your site was crawled with some kind of problem with the robots.txt and hasn't been re-crawled since.
-
Yes , Exactly
I am also worried For that only, Can you please help to identify my site problem
Thnx
-
That's very strange. The robots.txt looks fine, but here's what I see when I search for your site on Google.
-
Headers look fine and as you correctly said your robots and meta robots are also ok.
I have also noted that doing a site:www etc in google search is also returning pages for your site so again showing it is being crawled and indexed.
To all intents and purposes it looks ok to me. Someone else may be able to shed more light on the issue if they have experienced this error to this degree.
-
www.tajsigma.com This the Domain For Query robots 605- code
-
Just to check it that the live one, or just a test in GSC. Can you send a link to your site maybe in a PM.
-
Yes, It is Also okay there see Attached screenshot
-
Have you followed the following and in Google Search Console tried testing your robots file.
If you are allowing all, I would maybe suggest simply removing your robots.txt all together so it defaults to just crawling everything.
-
Thanx, Tim Holmes For Your Quick reply
But My robots.txt File is
User-agent: *
allow: /Also in All pages i have Add Meta-tag
Then After Page is Not Getting Fetched or Crawl by GWT.
Thnx
-
Hello Falguni,
I believe the error is saying pretty much everything you need to know. Your Robots file or robots meta would appear to be blocking your site from being crawled.
Have you checked your robots.txt file in your root - or type in http://www.yourdomain.com/robots.txt
To ensure your site is being crawled and for robots to have complete access the following should be in place
**User-agent: ***
Disallow: To exclude all robots from the entire server**User-agent: ***
Disallow: **/**If it is the a meta tag causing the issue you will require, or have it removed to default to the below.As opposed to the following combinations which could result in some areas not being indexed, crawled etc
_Hope that helps
Tim_
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Interest in optimise Google Crawl
Hello, I have an ecommerce site with all pages crawled and indexed by Google. But I have some pages with multiple urls like : www.sitename.com/product-name.html and www.sitename.com/category/product-name.html There is a canonical on all these pages linking to the simplest url (so Google index only one page). So the multiple pages are not indexed, but Google still comes crawling them. My question is : Did I have any interest in avoiding Google to crawl these pages or not ? My point is that Google crawl around 1500 pages a day on my site, but there are only 800 real pages and they are all indexed on Google. There is no particular issue, so is it interesting to make it change ? Thanks
Intermediate & Advanced SEO | | onibi290 -
Pagination & duplicate meta
Hi I have a few pages flagged for duplicate meta e.g.: http://www.key.co.uk/en/key/workbenches?page=2
Intermediate & Advanced SEO | | BeckyKey
http://www.key.co.uk/en/key/workbenches I can;t see anything wrong with the pagination & other pages have the same code, but aren't flagged for duplicate: http://www.key.co.uk/en/key/coshh-cabinets http://www.key.co.uk/en/key/coshh-cabinets?page=2 I can't see to find the issue - any ideas? Becky0 -
URL Parameter & crawl stats
Hey Guys,I recently used the URL parameter tool in WBT to mark different urls that offers the same content.I have the parameter "?source=site1" , "?source=site2", etc...It looks like this: www.example.com/article/12?source=site1The "source parameter" are feeds that we provide to partner sites and this way we can track the referral site with our internal analytics platform.Although, pages like:www.example.com/article/12?source=site1 have canonical to the original page www.example.com/article/12, Google indexed both of the URLs
Intermediate & Advanced SEO | | Mr.bfz
www.example.com/article/12?source=site1andwww.example.com/article/12Last week I used the URL parameter tool to mark "source" parameter "No, this parameter doesnt effect page content (track usage)" and today I see a 40% decrease in my crawl stats.In one hand, It makes sense that now google is not crawling the repeated urls with different sources but in the other hand I thought that efficient crawlability would increase my crawl stats.In additional, google is still indexing same pages with different source parameters.I would like to know if someone have experienced something similar and by increasing crawl efficiency I should expect my crawl stats to go up or down?I really appreciate all the help!Thanks!0 -
Affiliate & canonicals
Hi, any help with this one would be great.... www.example.com sells widgets online. They are also promoted on a 3rd party website www.partner.com. Currently www.partner.com links to a page on www.example.com that is completely branded with the 'partners' design, style and unique copy (you would think you were still on 'partner' website). I saw this interesting article from 2011: http://www.seomoz.org/blog/getting-seo-value-from-your-affiliate-links (in particular idea 1) Do you think adding a rel=canonical on www.example.com's partner page is still safe? All the best & thank you, Richard
Intermediate & Advanced SEO | | Richard5550 -
Places Listing in Search Results
Hi everyone, We have a company that hired us to set-up their Google Places listing for their 2nd location. The listing for the 1st location is very strong. Lots of reviews, Zagat rating, Knowledge graph, etc. In the search results the Google Places listing for the 1st location has merged with the website listing. You can see the link to the main site w/ a small grey google places listing directly below it. The client would like BOTH Google places listing to show up in the search results. They both show up on the map listing but not in the search results. Each location has its own listing in Google Places. We have also created different pages on the website for each location. Is there a way to get the search results to display places listings? I have noticed a few other business have done it by naming each of their multiple locations something slightly different. Then the search results seem to realize there are multiple locations and display the places listing in the search results. Anyone run into this? Any ideas? Thanks!!
Intermediate & Advanced SEO | | SeattleJoe0 -
Error 403
Hi SEOmoz community, Today, I checked the google webmaster tool of one of my clients, and ithere are 18 403 errors, I was wondering on how to fix those since it is the first time I come across these errors? How can I avoid that in the future? Thank you,
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
Image Search algo changes
Just seen the image algo changes that have apparently been put into place in the last 3 months My spam tester image site didnt seem to feel anything. Did anyone feel image search change (for better or worse?) And on what dates? Cheers Stephen
Intermediate & Advanced SEO | | firstconversion0 -
Search Engine Blocked by robots.txt for Dynamic URLs
Today, I was checking crawl diagnostics for my website. I found warning for search engine blocked by robots.txt I have added following syntax to robots.txt file for all dynamic URLs. Disallow: /*?osCsid Disallow: /*?q= Disallow: /*?dir= Disallow: /*?p= Disallow: /*?limit= Disallow: /*review-form Dynamic URLs are as follow. http://www.vistastores.com/bar-stools?dir=desc&order=position http://www.vistastores.com/bathroom-lighting?p=2 and many more... So, Why should it shows me warning for this? Does it really matter or any other solution for these kind of dynamic URLs.
Intermediate & Advanced SEO | | CommercePundit0
