Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO for E-Commerce Sites
Hi Everybody, I have two e-commerce sites just launched with not much content at the moment just user login pages for the clients to avail the service. The management is not interested to put much content there i think. Maximum what they will be putting only 5 pages of content in total, not more than this. Any practical tips how to optimize such sites especially when there is not much content. Best
On-Page Optimization | | Sequelmed0 -
Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare). We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain. If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains? Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error! Cheers, Dave.
On-Page Optimization | | davelane.verve0 -
What on-site issue could be causing Moz to not detect internal links?
Hey guys, We've done a crawl and none of our internal links are showing up. Are there any on-page factors that would prevent Moz from being able to detect our internal links? Thanks!
On-Page Optimization | | ATMOSMarketing560 -
Expert to Improve Site Speed Needed
Hey, I am looking for an expert who can improve the speed performance for my website NaturalSociety.com. Please please email me at mike@naturalsociety.com if you or your company works to improve website speed performance, or if you can recommend someone. Thanks in advance, -Mike
On-Page Optimization | | naturalsociety0 -
On Site Problem Caused Rankings To Drop?
I'm getting to the bottom of why my site dropped rankings on the 9th March. I noticed that in google there is a cached version of my site on 9th March at 8.37am which is when my rankings dissapeared. Presumably this is when google last crawled my site? I guess this means that google found something on the home page or on the site that it didn't like? I wonder if anyone can take a look and let me know if there's anything obvious. Could it be a duplicate content penalty as I have lots of categories pulling content from the same posts?
On-Page Optimization | | SamCUK0 -
Mentioning own site and keywords on here?
I have noticed that sometimes posters will talk about a site without mentioning what it is. I assume this is because it one of their clients so there is confidentiality, is there any other reason I should be aware of? its just that as I am new I am usually cautious and am considering posting my own site and mentioning all my keywords to ask for people’s verdict for my on-page SEO. Still working on it, will be ready soon, thought I would ask in advance. Regards,
On-Page Optimization | | Zoolander0 -
Sudden Site Rankings Drop
Good day guys, We have been following strict SEO strategies for the past 6 months, all sites have been improving incredibly well, all except one. The site in question is http://bit.ly/IH4pkM . The site is regarding automotive spray booth equipment. We were ranking on the first page for the keyword "spray booth" (which is the most important one), at place #4 for weeks on end. However since half-way last week, the site has been dropped to half-way the second page (#17). There are barely any crawler errors listed for our campaign on SEOMoz. There were several pages of which the meta description was missing, but that has been fixed earlier this week. When it comes to link building, I looked at what the top competitors were doing, and was looking for unique link building opportunities myself. We have received 0 webmaster tools warnings as well. I do not believe we are penalized due to the "penguin" update. After all, if you search for for the company's name in Google, it is still listed on there (# 2). Nor have we been part of dodgy link networks at all. So my question is, what do you guys believe made us drop the rankings? Is there some on-page issues I am overlooking? Any recommendations to restore out previous rankings? Kind Regards, Roderic
On-Page Optimization | | Michael-Goode0 -
Advantages of wordpress over dreamweaver
I have a site that I have built with dreamweaver (local service business not a lot of new content needing to be added) and I am curious how much of an advantage for me converting it to wordpresss would be. I see that cs5 works with wordpress and that lynda.com has a tutorial on integrating the two. I would like to integrate a blog into my site, but it is not totally critical that I do so. I would just like to hear what some of you experienced users have to say about the advantages of wordpress vs dreamweaver from an SEO prospective.
On-Page Optimization | | ayetti0