Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help recover lost traffic (70%) from robots.txt error.
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change. Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated. On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL? Thanks,
On-Page Optimization | | akin671 -
Best site Template, Structure, etc. for SEO
If I were to spin up a new site what do people recommend as the best template, services, etc. Do you have an example of the perfect structure, I want to point my team to an example page and say - This is perfect, do this but for our product (structure, content amount, etc) Thank you,
On-Page Optimization | | Jamesmcd030 -
Tracking down rel="canonical" on Wordpress site
A rel="canonical" is being added to every page and post on my Wordpress site - not tag results, not category results. Not a major problem, right? Except that I don't know where it's coming from. I've tried tracking it down - change the theme back to a default one, turn off all the plugins - it's still there. Is it coming from .htaccess perhaps? The only issue it is causing is that it has causes me to have to turn off the canonical option in Platinum SEO as that was resulting in two identical rel=canon on each page. It doesn't seem to be causing problem but I'd like to get a better understanding of what it going on.
On-Page Optimization | | robandsarahgillespie0 -
Site Not Ranking for Key Term
Question for my fellow Mozers I have a ranking question that I cannot put my finger on. I have a site (visitplano.com) where the client wants to rank for the keyword "Plano". I can't say if the site was previously ranking for this keyword, but I looked into the basic SEO practices and found that the keyword is incorporated in: Domain Title Content There is a lack of internal linking and anchor text within the content External links - 1,558 DA - 46 PA - 55 Currently, the website does not rank for the keyword "Plano". Could someone shed some light on why they aren't ranking or what I may be missing? I would greatly appreciate your help.
On-Page Optimization | | flcity150 -
Multiple Organization Schema on the same site
I creating a preferred supplier list on my site and wanted to use the Organization Schema for the company details. Is there a issue with having more than one org schema on the same site? or should I just use the one for my company. Thanks in advance
On-Page Optimization | | gregdicksonuk1 -
Competitor Out Ranks My Site
Hi This is my first question on SEOMoz and I am confused how a competitor can out rank me for a local search term SEO Tamworth. I have a higher Page Authority, Domain Authority, Number of links etc.. Would this be down to the quality of the links to my site or could it be down to the structure. I score an A for SEO on the on page analysis. Regards James
On-Page Optimization | | Cocoonfxmedia0 -
Should I link my 3 E-Commerce sites?
Good Morning, I have 3 E-Commerce sites that all sell the same products, but have unique content on them (unique text, unique urls, same products).... Up until now, I have not put any links from one to any of the others... I just started to wonder about that since these are all related to the same industry, and are owned by 1 company, what would be the downside to linking them... Does anyone have any advice on if I should link to each site from the other 2 sites? Also, if you think I should be linking them, please advise how you would do it (on which pages, how many links, anchor texts, etc...) Thanks a lot!
On-Page Optimization | | Prime850 -
Remove internal site SERPS from Google Index?
1. Internal Serp pages did not have a robots meta tag 2. As a result, client site has thousands (~4,400) of internal site SERP pages in the Google index. 3. We added the NoIndex, Follow attribute to all internal SERPS 4. We Disallowed: domain.com/internal-search-operator in Robots.txt 5. No new SERP pages are being indexed, but the other 4000 something that were already there are still in the index weeks later. 6. The pages are dynamically created and still work, so I can't use the Remove Content tool from google, because the pages don't 404. Is there any way to get these pages out of the index besides just waiting and hoping google eventuall drops them? Thanks
On-Page Optimization | | delegator.com0