Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
urmiawebseo site urmia
Re: Star snippet not work
On-Page Optimization | | gercaarre3
Hello I write a json script for star snippet in my website but not work in my any page. can see it in this address https://www.urmiawebseo.com when I googled my keyword "طراحی سایت در ارومیه" my site dont show. #on_page0 -
IMG TITLE no longer used in Wordpress?
Err, when did this happen? Although you can still enter a Title in the WP image library, when inserting into a post the Title tag is not implemented. Alt text is there OK but not Title. There was a plugin around that restored this basic seo functionality but it's no longer available. I mean - honestly!
On-Page Optimization | | robandsarahgillespie0 -
Should I noindex our WordPress Categories?
What are the conditions under which we should noindex blog categories? I'm asking because I want to decided whether or not to noindex our categories. here.
On-Page Optimization | | Heydarian1 -
Latest news in site-wide footer?
I've asked my web team to add a section to our home page to showcase our latest news and content (articles, blog entries (same domain), white papers, case studies, etc.) I think it will be a good user experience and also provide an opportunity to keep a fairly stagnant home page updated with fresh content. My web team wants to add the section to our footer, which is site-wide. I'm sure this isn't a bad user experience, but will it cause problems or eliminate any SEO value (if any) for this regularly updating content?
On-Page Optimization | | mbrusyo0 -
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
WordPress Crawl Errors
I recently added wordpress to my site and get the following errors: Duplicate Page Content http://agrimapper.com/wordpress/ http://agrimapper.com/wordpress/index.php How do I define the canonical page on a .php. 4XX (Client Error) http://agrimapper.com/wordpress/index.phpindex.php Any ideas where the 4XX error comes from. Thanks.
On-Page Optimization | | MSSBConsulting0 -
What is the best out-of-the-box optimized WordPress Theme?
I've reviewed other answers on this forum and searched the web. Still looking for a good answer that's actually specific and can explain why. I've looked at studiopress.com themes, pagelines.com which I love but doesn't appear to have much optimization out of the box, http://themeforest.net/item/udesign-wordpress-theme/253220 http://www.vklabx.com/wordpress-themes/wordpress-theme-ad-flex-niche.php PS - not looking for answers that involve the use of SEO plugins, but if there's a theme you know of that takes minimal editing to improve optimization I'd be interested as well. Thanks in advance, hope this can help someone else as well
On-Page Optimization | | Aqua0 -
Ranking Issues Recently Popping Up
We have a site that based on your research tools, holds its own in almost all aspects in regards to # of links, # of different linking domains, quality of links, mozrank, moztrust and all that stuff. Compared to our top competitors, we do very well based on your tools via our campaign monitor. The issue is we seem to be dropping like crazy every month in our rankings and traffic despite this fact, and we can't get our head around the cause. I do have a couple of ideas, and I wanted to run them by you guys to get your opinion. Domain: bonitaj.com My Thoughts On Possible Issues... 1. Text Content & Panda Update I know one of the big things with the panda update was quality of content. I know one thing we have for sure is a lack of "text-based" content. Sure, we have home page, main cats, sub-cats and product pages, but they are mostly just windows into the product pages, and don't have a whole lot of good copy. THIS IS MOST EVIDENT ON OUR PRODUCT PAGES, where each product page is loaded with content, but only 3-4 very short paragraphs of text. Do you think this is hurting us? THE ONLY ISSUE IS THAT our competitors also don't have a whole lot of text-based content on their pages. 2. Too Many Category Pages & Same Products Featured Somewhat on them I think another problem may be that on each category page, we do have a lot of the same products featured. I don't think its crazy duplicate content or anything, but I do think that back in the old days we got a little crazy with creating "niched out" category pages that pretty much feature the same products as some of the more important and base category pages. Do you think this is hurting us? I've pitched a solution to this that involves trying to tone down the amount of sub-cats we feature that were originally geared towards attracting long-tail traffic. In the end that really isn't working anymore anyway, so maybe we're spreading our site thin by going to deep with some of these niche category pages? 3. Lack of a sitemap? We used to use an xml sitemap, and really don't anymore. We have nothing on file with google webmaster tools. I've recently read in one of your blog posts that a simple thing like adding a good sitemap could help our 600+ page site or so get crawled a bit deeper allowing more pages to rank? IN THE END, MY QUESTION IS SIMPLY, IF THERE IS ONE OR TWO THINGS I CAN DO TO GET OVER THIS HUMP, WHAT WOULD YOU SUGGEST?
On-Page Optimization | | AarcMediaGroup0