Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Wide Links
Howdy Moz! So our agency has been around for long enough to have a few sites we've built that have our credit in their footer resulting in a site wide link. Mostly just our name. We've heard that Google does not particularly like site wide links, should we go through and remove some of these old links?
On-Page Optimization | | wearehappymedia0 -
Does no-follow for pages affect site ranking?
Hey, I have a question. On my site, it's divided into the main site and the blog is in a subfolder of same domain. Within the main site (same domain), there are MANY checkout pages and other internal pages we use though all with "NO FOLLOW" on each. Despite it having "NO FOLLOW", will it affect our blog rankings in any way or domain ranking?"
On-Page Optimization | | Mirian0 -
Best schema.org wordpress plugin?
Dear all,
On-Page Optimization | | plcarrara
Im doing SEO for a finances website, and Im struggling to find a schema plugin for wordpresss that would contain "News Article" schema. Can you give me a hand with this? Best regards. Pablo0 -
Problemas with my htaccess file
Hi all, I have two doamins actived which content is the same. Regarding Google some of you guys told me that I wouldn't be penalized but If I wanted to do a redirection 301 it should be done from one domain to the domain which is my main market. And I just did it. But the problem is that although I have made the corresponding modifications in the htaccess file it doesn' work. When I write www.piensapiensa.com goes to piensapiensa.com (as I configured the WMT) and not to www.piensapiensa.es, in which my market is mainly present. Heres the code for the HTACCESS file: Options +FollowSymLinks +ExecCGI
On-Page Optimization | | juanmiguelcr
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www.piensapiensa.es [NC]
RewriteRule (.*) http://www.piensapiensa.es/$1 [R=301,L]
<ifmodule mod_rewrite.c="">RewriteEngine On # uncomment the following line, if you are having trouble
# getting no_script_name to work
#RewriteBase / # we skip all files with .something
#RewriteCond %{REQUEST_URI} ..+$
#RewriteCond %{REQUEST_URI} !.html$
#RewriteRule .* - [L] # we check if the .html version is here (caching)
RewriteRule ^$ index.html [QSA]
RewriteRule ^([^.]+)$ $1.html [QSA]
RewriteCond %{REQUEST_FILENAME} !-f # no, so we redirect to our front web controller
RewriteRule ^(.*)$ index.php [QSA,L]</ifmodule> Thanks in advanced.0 -
Why is this site 1st in Google??
Hello The site www.woodensigns.net is 1st in google for the keyword "wooden Signs". All seo indicators are poor except keyword in url; wich, i thought, was not a + for google anymore. Could someone help me to understand here? Thank you Emmanuel
On-Page Optimization | | manu450 -
Site structure question
I'm currently working on a very awkward custom-WP setup, in which I can't maintain the present drop-down navigation menu without having those pages under a parent or without completely recoding everything. I have two requirements, for SEO purposes I'm looking for the following structure for each targeted landing page: www.example.com/landing-page as opposed to www.example.com/sub/landing-page Of course, having my landing pages as a child, I get the latter of the two. For navigational purposes they need to fall under a specific category in a drop-down menu. With any other theme or setup this is an easy fix, but not here. What I have now is that the landing pages are currently placed under a parent category page. But, they have custom permalinks. The permalinks are setup as follows www.example.com/landing-page But, technically the exact structure is still www.example.com/sub/landing-page which then redirects to the custom permalink. So, my question is - in an attempt to get my most important landing pages close to the root for better PR and crawlability, do I still get the same benefit with my current setup? Is this structure I have, better, worse, or indifferent? Thanks.
On-Page Optimization | | JayAdams320 -
Canonical Tag for Ecommerce Site
My client has an ecommerce site with over 1,000 products. We have a ton of duplicates because of how their ecommerce system handles product pages. Each time a new product is added, there is a default product page created (/product/12345-product-name.aspx). Each time that product is added to a specific product category, another, separate URL is created (/product/office-chairs/12345-product-name.aspx). The site has over 1,000 duplicates (at least one for each product) because of how the ecommerce system structures URLs. We are unable to have unique content on /products/12345-product-name.aspx and /product/office-chairs/12345-product-name.aspx because both pages pull from the same database. Their webteam informed me that they can't implement canonical tags on individual pages, they must be dynamically added to the site all at once. Thus forcing me to choose all of the default product pages as primary URLs. Both types of URLs are getting indexed and the product URLs that were added to the categories are SEO friendly so I'm leary to eliminate one or the other with a canonical tag or a no index. Suggestions?
On-Page Optimization | | DynoSaur0 -
Anchor Texts on Internal Links on my Site Question
[Hey folks, Anchor text question for internal links on my site. We all know that proper anchor text is important for rankings, both in links pointing to your site and links on your site. Google webmaster tools is showing my top internal anchor text links to be letters. This is because of navigation on the site which is in a global footer on hundreds of thousands of pages. Do you think that this dilutes or minimizes the relevance of the real anchor texts that we are targeting which trail behind these, or if this went away it would have no effect on the rest of the anchor texts on the site. How would you handle this, we can just get rid of this navigation because it's passing PR and helping the pages get spidered. Thanks in advance. Anchor text 1. g 2. q 3. d 4. v 5. k 6. t 7. o 8. u 9. a 10. p 11. m 12. r 13. e 14. x 15. y 16. l 17. i 18. z 19. c 20. j 21. s 22. b 23. f 24. h 25. w 26. n](mailto:acaldarea@hotmail.com)
On-Page Optimization | | irvingw0