Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a benefit to hiding words from Google on ecommerce sites?
I think we all know that ecommerce sites have a lot of repeating functional texts on them. Is there a benefit from hiding this text from Google's crawler? Take this page for example, http://storage3.static.itmages.com/i/16/0805/h_1470425505_1090542_224cc344d4.png Some of the most dense words on the page are "Add to cart", "Add to Wishlist", "New", and "Sale". Is there any benefit to hiding those words from Google? The method of hiding I am talking about is not cloaking, but this, https://www.google.com/support/enterprise/static/gsa/docs/admin/70/gsa_doc_set/admin_crawl/preparing.html#1076243 using the google:off index tag. So the content will be there still, but it will not be indexed.
On-Page Optimization | | LesleyPaone0 -
Help! A site has copied my blog!
My site tanked on July 21 and I have been working so hard to bring it back up but nothing is working. Today I looked at "Links to Your Site" on Google Webmasters and I see a copy of my site on another URL. mysite.eemovies.org/mycategory/mypost The domain name is eemovies.org and then all my stuff is wrapped around it and all my content is there! How do I stop this?!
On-Page Optimization | | 2bloggers0 -
The correct way to go from PHP site to HTML site?
I have a website fully coded in PHP and I am doing a re-design over to an HTML site. I searched through the Q&A and there were some conflicting answers. Some said you will need to 301 all the pages. Others said to use the .htaccess to parse all the files as html. What is the correct way I should go about this? Thanks in advance!
On-Page Optimization | | reliabox0 -
Wordpress Title Element tag too long?
So apparently when I add the name of the blog within Wordpress, the name of the blog shows up after the the main title of each blog post and page. So trying to keep the title element >70 characters, how does one name the Wordpress blog as mine is/was: Lake Tahoe wedding photographer / Heidi Huber Photography and keep that only on the intro page and leave each page of the blog and the titles of the blog posts to not have "Lake Tahoe wedding photographer added to the end of the Title Element? Anyone have any idea? I also had gone in at one point and updated every single Title on each page/post using Yoast's SEO and the SEOMOZ crawl is still coming up with over 216 caution errors for the title page. Thank you so much for any advice! Heidi
On-Page Optimization | | hmhuber0 -
Wordpress pages URL's redirection.
I was checking W3C Markup Validation and in report it was shown that that pages (not post or any other URL's just PAGES) at investmentcontrarians.com are 301 redirected. e.g. original URL "http://www.investmentcontrarians.com/debt-crisis" which is redirected to "http://www.investmentcontrarians.com/debt-crisis/" I know that its not that serious issue, but still want to know why only pages are being redirected and how can we avoid it.
On-Page Optimization | | NumeroUnoWebSolutions0 -
Duplicate Title & Content in WordPress
I'm getting a lot of Crawl Errors due to duplicate content and duplicate title because of category and tag posts in WordPress. I rebuilt the sitemap and said to exclude category and tags, should that clear up the issue? I've also went through and did NO INDEX and NO FOLLOW for all categories and posts. Any thoughts on this issue?
On-Page Optimization | | seantgreen0 -
Site Structure
I'm confused about the best way for seo to set up the site structure . i understand the examples of the pyramid diagrams and how link juice flows, however does this mean that global navigation is not good? It appears the pyramid structure leads to the designated number of category pages (we'll use five) and they lead to the 5 content pages etc and some "superman pages" can be linked to from the home page but is this is global navigation or anchor text navigation and is gloval navigation acdeptable for content pages? Any input would be greatly appreciated. Thank you.
On-Page Optimization | | JulB0 -
Wordpress Pages vs. Posts
When building a blog to promote a particular affiliate offer, I usually like to use a static "page" homepage and then have my posts displayed on another part of my site. I've noticed that my wordpress pages almost always rank higher than wordpress posts and I can't explain it... Here are some possibilities I've thought of: XML Sitemap priority is set at 60% for pages and only 20% for posts My main navbar lists the pages which consequently means they get linked to on every page on my website. Some other phenomenon within the wordpress framework... If it helps, I use Thesis v1.8 on all my sites. I guess my ultimate question is: If pages do in fact rank higher than posts, is it worth it for me to go back and change the site structure on all my blogs which are using posts instead of pages. I know making major modifications like that can be disasterous but will it ultimately pay off? Thanks
On-Page Optimization | | drewhammond1