Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Issues with Multiple H1 tags on homepage?
Hi folks, My homepage has 3 identical H1 tags due to the fact that I have had to create individual hero images (with headings) for desktop, tablet and mobile. I couldn't get my theme to display the layout in exactly the way I wanted on each device without doing a specific hero image and tag for each device type. Does this have a major impact on my SEO? Thanks,
On-Page Optimization | | Veevlimike
Mike.0 -
Site no indexed after a week loadbalancer, cache-control?
hi All apologies to everybody in advance my level of SEO or technical stuff is quite minimal. we have setup a new site with a top level domain sitting on multiple servers ( https.www.mysite/new/ on one and http.wwww.mysites/old/ on another) different pages,content etc. it's definitely a patchy solution until what is left in the old server will be migrated to the new one. however the new site is still not indexed after around 2 weeks. I have checke on moz and and google console ( fetch and rendering) and nothing seem to block indexing. no issue with robot.txt ( either on https.www.mysite/new/robots.txt or http.wwww.mysites/old/robots.txt) meta-robot. any idea on what could cause the problem? can it be an issue with the cache-control set as : no-store no-cache? Or the loadbalancer that is preventing google to access ? Dario
On-Page Optimization | | Mrlocicero0 -
Blog issue broken link
Taking Great Photographs Underwater May 25, 2015 By sdwellers@aol.com No comments yet florida keys, key largo diving Excuse my ignorance, I suspect this is an easy issue...but at the the top of each of my blog posts have what you see above....the "No Comments yet" tab is showing as a broken link 404 error...?Why? And how to fix?Thank you
On-Page Optimization | | sdwellers0 -
Photon for Wordpress experience
Has anyone tried switching on the Photon for Wordpress CDN/image caching feature? If so, did you experience any problems and have you noticed any improvements?
On-Page Optimization | | pugh0 -
Is a site map necessary or recommended?
We have a website that has been up for the past 4 years without a site map. Google is indexing it. Do we need a site map? Do you recommend we create one and submit it to goggle and bing? The site is www.logobids.com Thank you.
On-Page Optimization | | IsaacH0 -
Changing my site (dramatically)
I am about to do a complete site change. I am going to WordPress. I am ranked #2 on SERPS. Will I lose rank for changing everything on my site? I have 500 pages indexed but I am about to have 30k indexed. It is a real estate site that is switching from a "framed" solution, to a listing indexed solution. If I make good use of my keywords etc (on site optimization) will I be at risk of losing risk just for changing my site?
On-Page Optimization | | JML11790 -
Mozbar # issue
Hi, When viewed through the Mozbar page analysis tool my Joomla site is always displaying a bullet point and space character before the site. Is this likely to effect search engine reading of my ? www.clearvisas.com - is the site. Please have look and let me know what you think. If you are a Joomla expert your feedback on my website and this issue will be really welcome. Thanks, Fuad
On-Page Optimization | | Fuad_YK0 -
How to make google not index quotes from other sites?
Hey guys, I have a site where we post quite a lot of info from other sites. We don't want google to de-index our pages because parts of it are quotes from other sites. What would you use to make it so Google sees it's a quote from another site? Or to just make Google not index the quote? Thanks!
On-Page Optimization | | StefanJDorresteijn0