Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question on site structure
My client is a nationwide company. They provide building maintenance services in 7 different cities. In each city they provide a different range of services. They currently have a single service page for each service and no mention on that page of the cities they offer the service. The service pages are getting no SERP visibility. We are running Paid Search and recommending SEO. I'm wondering whether it would be beneficial to build out specific service pages for each city so the content is more relevant to both users and search engines. What is best practice in this situation? Client wants to dominate SERPs in each market for the services they offer.
On-Page Optimization | | SEOinSunnyNelson0 -
I am trying to better understand solving the duplicate content issues highlighted in your recent crawl report of our site - www.thehomesites.com.
Below are some of the urls highlighted as having duplicate content -
On-Page Optimization | | urahul
http://www.thehomesites.com/zip_details/76105
http://www.thehomesites.com/zip_details/44135
http://www.thehomesites.com/zip_details/75227
http://www.thehomesites.com/zip_details/94501 These are neighborhood reports generated for 4 different zip codes. We use a standard template to create these reports. What are some of the steps we can take to avoid these pages being categorized as duplicate content?0 -
Latest news in site-wide footer?
I've asked my web team to add a section to our home page to showcase our latest news and content (articles, blog entries (same domain), white papers, case studies, etc.) I think it will be a good user experience and also provide an opportunity to keep a fairly stagnant home page updated with fresh content. My web team wants to add the section to our footer, which is site-wide. I'm sure this isn't a bad user experience, but will it cause problems or eliminate any SEO value (if any) for this regularly updating content?
On-Page Optimization | | mbrusyo0 -
Best wordpress theme seo 2014
In your opinion what are the best wordpress theme in 2014 that are responsive, good speed and seo buildup
On-Page Optimization | | maestrosonrisas0 -
How to Recover From Unstable Site Structure History
I have a site that has suffered several phases of restructuring. Apparently its owners were unsure as to which direction to take when it came to structuring their content and URL schema and subjected the site to several rounds of poorly thought through implementations (i.e. example.com/content/page-title, example.com/page-title, example.com/"silo"/page-title, etc.), all within a 8 month period. I posted the originating question here on this Q&A Forum. I want to thank EGOL and Cody for taking a stab at it. What would be a good strategy to help a site like the one I describe above begin ranking again?
On-Page Optimization | | UplinkSpyder0 -
What is this site all about?
Can anyone here tell me what this site is all about?... I really need to know, i have a blog.
On-Page Optimization | | Wales33030 -
Keywords vs Tags In Wordpress
Whats the difference between Keywords vs Tags In Wordpress? Also sine meta keywords don't matter in a website do they matter on a blog?
On-Page Optimization | | splashmedia0 -
Best way to link to an affiliate site.
I have a site that promotes a single affiliate product. Would it be better to have a "buy" button in the content that goes straight to the external affiliates website or should I send it all to an internal page that redirects thus only having 1 external link to the product site.
On-Page Optimization | | jafabel0