Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Way to Handle Multi-Language Sites
In the last year we've made a few significant changes to the structure of our site - namely adding translations for a few languages. We have historically been gaining in organic search by about 10% each month, but in the last two months we've leveled out and seen a slight dip. I am wondering if this has something to do with the addition of the second language, and namely if there's a chance we've been penalized due to duplicate content. We have almost all pages / content on the site translated by a translator, but the way the development works the site will grab the english version if a translation hasn't been added - potentially adding some duplicate content? The URL structure remains the same, other than the addiion of the language - site.com/our-tour vs site.com/de/our-tour We also haven't translated the tour name itself, so that remains the same. Just wondering if anyone has any feedback on best practices here or things I should be looking out for. Thanks in Advance.
On-Page Optimization | | mkgreyound1 -
Wordpress / Full URL In Menu Box
I came across an article online (not Moz) that says adding the full url in a menu is a Google standard. So when you make a menu link you put "www.example.com/page" instead of "/page". What are your thoughts on this? Any real reason to? Y840pbN lrwZPDj
On-Page Optimization | | InfinityTechnologySolutions0 -
Blog issue broken link
Taking Great Photographs Underwater May 25, 2015 By sdwellers@aol.com No comments yet florida keys, key largo diving Excuse my ignorance, I suspect this is an easy issue...but at the the top of each of my blog posts have what you see above....the "No Comments yet" tab is showing as a broken link 404 error...?Why? And how to fix?Thank you
On-Page Optimization | | sdwellers0 -
Site appears to rank very low
Hi, A site we manage is ranking very low for it's main key phrases. The site is www.moremouse.com. For example for the phrase "orlando vacation rentals" it ranks around page 12 which seems very low considering the DA, PA, links, etc. compared to many sites ranking much much higher. Can anyone see anything obvious that is causing it to rank so low? Thanks Pete
On-Page Optimization | | QbicIS0 -
How to handle wordpress tags
Hi fellow SEO mozzers, I am getting 'duplicate content' errors when our site is crawled, mainly down to our WordPress blog and how we have handled tags. Currently, they are being crawled and as such are regarded as duplicate pages. I have read several different articles on how to handle tags. Some suggest noindex the tag URL's. Others suggest to optimize them and allow them to be indexed since Google has confirmed they won't penalize a WordPress site for having archive pages that publish and point to the same content. It will select the best link to represent the cluster of links. Over the past few months, nearly 4% of our WordPress traffic have been referred by tag pages listed in search engines. Initially I was going to noindex the tag pages, but going on the above info I wonder should I leave them as they are? Or is the issue that having duplicate content will lead to inefficient crawling? Any views/opinions on how best to handle this?
On-Page Optimization | | efink0 -
SEO for New Magento Site??!?
Hi All, We have had a new site delivered by our developers (hurrah...the old one was to terrible for words) but we seem to be having a lot of issues with the new Magento platform. What they have done is used the community version and tried to customize it. I came on to this project about a month into the build and although Magento does seem to do a lot things well there do seem to be some problems. From an SEO prespective we have seen some increase on some search terms and a drop of in others. I would be interested in hearing from other Magento users about their experiences with this platform and any ideas on how to crank up the activity. Our site is at www.nationwidepharmacies.co.uk . There are few odd bits including the side navigation which seems to be very clunky and not overly customizable in this version. Any useful criticism would also be well received. Look forward to hearing Nic
On-Page Optimization | | nicc19760 -
Improve Site SEO
Hey Guys I was wondering if you could give me advice to improving my SEO. I have used all the tools on this site and have improved it a lot already, but wondering if there is any glaring issues, that I could fix now www.treelifedesigns.com
On-Page Optimization | | treelifedesigns0 -
Duplicate content Issue
I'm getting a report of duplicate title and content on: http://www.website.com/ http://www.website.com/index.php Of course, they're the same pages but does this need to be corrected somehow. Thanks!
On-Page Optimization | | dbaxa-2613380