Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: / -
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare). We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain. If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains? Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error! Cheers, Dave.
On-Page Optimization | | davelane.verve0 -
Is this canonical issue?
WP site has automated canonical on domain.com/ by Yoast plugin but there is URL (/?page=kontaktine-forma) where same domaim.com/ canonical tag was put on. I made 301 redirect to main page. Is this a good practice?
On-Page Optimization | | OVJ0 -
Increasing pageviews on a social networking site
Our social networking site, Yookos, has a target of 25 million pageviews by yearend 2013. This is part of my responsibilities. As with all social platforms, user authentication is required before one can access the content. What can i do to improve the pageviews exponentially?
On-Page Optimization | | seoworx1231 -
Issues with Product Pages Getting Index In Google
I just started working here the other week and one of the big issue is that a lot of the product pages are not getting index in google. We have an xml.gz site map they submitted a long time ago. My guess is it might be something with not enough content on the pages? Here are a few example of pages that are not getting index in google. http://www.rockymountainatvmc.com/p/43/-/439/716/-/33097/Alpinestars-Dual-Motorcycle-Gloves http://www.rockymountainatvmc.com/p/47/-/201/803/-/28948/Camelbak-Blowfish-2013 http://www.rockymountainatvmc.com/p/46/-/203/836/-/6996/MSR-Head-Case http://www.rockymountainatvmc.com/p/44/54/208/764/80/1220/Galfer-Brake-Pad-Sintered-Metal There are 100's that are not indexed just trying to figure out what we need to do! We are working on new content to them all but we have over 5000 products so it will take a long time. We also have the reviews on the pages and are looking at starting a Q&A on page to help get more unique content.
On-Page Optimization | | DoRM0 -
Problem with Plugin: Wordpress SEO by Yoast
Hi Hi, I'm using the Plugin "Wordpress SEO by Yoast". Unfortunatelly, this plugin generates a sourcecode looking like this: name='description' content='XXXXXX.'/> Seomoz can't read this and tells me, that my pages have no description. Google doesn't matter and reads it correct. Is there anything I can do about this? Many greetings André
On-Page Optimization | | waynestock0 -
Linking within Secondary Site
So we've got a secondary site that has quite a bit of authority & links that used to have all types of info on parasailing. All those pages are gone and homepage is now a salespage (management decision, not mine) Our main site sells a wide range of tours and activities and does have a page for parasailing. The secondary site uses the same template/navigation as our main site (again, not my decision). Do you think that's an effective way to send link juice to our main site? The secondary site has some pretty awesome high authority sites linking to it. I've considered 301'ing the whole site to our main site but it's got a really solid domain name and I'd like to take up 2 SERP listings (main and secondary site) Is there a better way to have double listings but still send a good amount of link juice?
On-Page Optimization | | SoulSurfer80 -
What is the best out-of-the-box optimized WordPress Theme?
I've reviewed other answers on this forum and searched the web. Still looking for a good answer that's actually specific and can explain why. I've looked at studiopress.com themes, pagelines.com which I love but doesn't appear to have much optimization out of the box, http://themeforest.net/item/udesign-wordpress-theme/253220 http://www.vklabx.com/wordpress-themes/wordpress-theme-ad-flex-niche.php PS - not looking for answers that involve the use of SEO plugins, but if there's a theme you know of that takes minimal editing to improve optimization I'd be interested as well. Thanks in advance, hope this can help someone else as well
On-Page Optimization | | Aqua0 -
On-page optimisation for CMS based sites
With so many sites being based on a CMS, and with so many hundreds of different CMS out there, as SEO consultants how do you recommend dealing with on-page optimisation for a client where you discover their site is built with a CMS you have not previously used (or even heard of!)
On-Page Optimization | | bjalc20110