Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Random /feed 404 error from a wordpress site
My Moz Analytics report shows a 404 error on a page which I think should not exist at all. The URL is http://henryplumbingco.com/portfolio-item/butler-elementary/feed/. When I checked webmaster tools, it looks like there are a number of random /feed urls throwing 404 errors. I am using WordPress and the Enfold theme. Anyone know how to get rid of these errors? Thanks,
On-Page Optimization | | aj6130 -
Does anyone know of an api for on site SEO?
I have searched for one, but really cannot find one that fits my needs. I am looking at making an on site grader / service that will check pages and point out SEO problems. One that I have found that I like is seorch.eu but they do not have an api. I do not want to reinvent the wheel if I do not have to. Also, the api does not have to be free, or it does not even have to be an api, it can be a self hosted application too.
On-Page Optimization | | LesleyPaone0 -
Photon for Wordpress experience
Has anyone tried switching on the Photon for Wordpress CDN/image caching feature? If so, did you experience any problems and have you noticed any improvements?
On-Page Optimization | | pugh0 -
H1 tag in wordpress
Hi there! I'm having problems to locate the h1 tag in wordpress (in each page) in order to optimize the pages for SEO. When introducing text it starts with h2. Where can I locate it? Thanks in advanced.
On-Page Optimization | | juanmiguelcr0 -
Canonical Tag for Ecommerce Site
I implemented a canonical tag on each product page for my clients ecommerce site and my rankings tanked. Has this happened to anyone else? If so, when can I expect rank to return?
On-Page Optimization | | DynoSaur0 -
Good Internal Site Structure Idea?
Hello SEOMoz, After reading a bunch of your Site Structure articles, I've decided to make ours more flat. There are numerous pages on our site which are linked to directly from our homepage, wasting mysterious amounts of Link Juice every day. I want to remove most of these links so that the Fewer, and now more heavily weighted, Homepage Links will be more powerful... but I am worried that the pages which I am knocking down to the 3rd tier level already have high rank and are distributing this Juice to other pages. The problem is that 3 of these 9 pages are great for assisting our sales team, so I cannot take those 2 links off of the homepage...so I will be forced to Nofollow them instead. I am worried this is cutting down the number of pages on the site, also cutting out content which was previously indexed. Is this whole thing a good idea at all? And should I just leave those 2 pages alone because I can't remove the link? I'm thinking maybe I should rel=canonical it back to the homepage? I am ultimately trying to rank the homepage for the keyword "POS Software" and this is my on-site strategy for it. Maybe adding a link from those 2 pages that say "POS Software" back to the homepage is the best bet in this scenario? I am trying to learn the absolute best thing to do instead of guessing. Thanks! Derek
On-Page Optimization | | DerekM880 -
Home page duplicated content issue
Hi there! The home page of my site can be seen under www.mysitename.com and www.mysitename.com/EN/ or www.mysitename.com/ES/ (depending on your language). I understand that this is duplicated content because they show the same content under different URLs. To solve this we've done (depending on your language) a 301 redirect from www.mysitename.com to www.mysitename.com/EN/ or www.mysitename.com/ES/ Is this correct? Thanks!
On-Page Optimization | | Xopie0 -
Is there any benefit in on-site duplicate content?
I have about 50 internal pages on my site that I want to add a "Do it yourself tutorial" to in an effort to build the quality of the pages. Is this going to de-value the content if I put it on all 50 pages? It's difficult to write similar content 50 different ways.
On-Page Optimization | | BradBorst0