Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Two Day After Starting Moz Pro campaign i experienced Sudden Huge Traffic Drop and My site ranki Drops in my site..Please I need your help
Two Day After Starting Moz Pro campaign i experienced Sudden Huge Traffic Drop and Drops in my site..Please I need your help
On-Page Optimization | | zizutz0 -
To avoid the duplicate content issue I have created new urls for that specific site I am posting to and redirecting that url to the original on my site. Is this the right way to do it?
I am trying to avoid the duplicate content issue by creating new urls and redirecting them to the original url. Is this the proper way of going about it?
On-Page Optimization | | yagobi210 -
On Site Errors
HI Folks I'm monitoring a small Australian site bluetea.com.au . Currently I have a SEO specialist who does month onsite maintenance work for this site. However each month I continue to see errors in webmaster tools... as an example, currently webmaster tools suggest we have 21 short meta dic and 26 duplicate title tags..... Examples given are Short Meta Discp /cleaning-products-and-our-health/toxic-cleaners/ /colour-consultant/fushia-door/ /portfolio/parisian-apartment-black-kitchen/parisian-apartment-black-kitchen/ Duplicate title tags <a id="zip_0-anchor" class="zippedsection_title"></a>concrete-kitchen | Blue Tea Kitchen Designs
On-Page Optimization | | PHD/kitchen-trends-and-material-innovation/concrete-kitchen-2/
/kitchen-trends-for-2013/concrete-kitchen/<a id="zip_1-anchor" class="zippedsection_title"></a>Potts Point Kitchen | Blue Tea Kitchen Designs
/portfolio/potts-point-kitchen/
/portfolio/potts-point-kitchen/pott-point-kitchen/My SEO tells me that he has solved all these issues but after one or two months they still remain in webmaster tools... can anybody help me understand why?Thank you
0 -
Keyword Stuffing Issues
Hi Folks, At this URL https://purplegriffon.com/courses/project-management/prince2 as an example, you can see that I am outputting all course dates in a table. Each table row contains information regarding the specific course such as start date, duration, location but also course title. In the course title it will obviously contain the keyword, in this example that is 'prince2'. So my question is this. As the MOZ on-page grader indicates that I am keyword stuffing (even though I receive an A grade) will Google be clever enough to know that I am displaying a table of events for PRINCE2? Or will it class the page as containing too many keywords? Each event links to a specific event page for that event, so is unique. I am interested in hearing the thoughts of the community on this. Thanks. Regards Gareth
On-Page Optimization | | PurpleGriffon0 -
Redirecting pages (old site to new site)
I have a question- there is one location, one set of pages for both the old and new site on the same host environment so when I did the redirect it get into a loop trying to redirect from itself to itself Not sure how its gonna affect SEO. Will pages get hit for duplicate content?
On-Page Optimization | | Yanez0 -
Need suggestion: Should the user profile link be disallowed in robots.txt
I maintain a myBB based forum here. The user profile links look something like this http://www.learnqtp.com/forums/User-Ankur Now in my GWT, I can see many 404 errors for user profile links. This is primarily because we have tight control over spam and auto-profiles generated by bots. Either our moderators or our spam control software delete such spammy member profiles on a periodic basis but by then Google indexes those profiles. I am wondering, would it be a good idea to disallow User profiles links using robots.txt? Something like Disallow: /forums/User-*
On-Page Optimization | | AnkurJ0 -
Wordpress category links not working
Hi All of sudden, my category links are not working. Any tips on figuring out what's causing this? Looks like permalink problem with newer wordpress version. I turned off all the plugins see if this cause any problems. Still not being able to find any option. Here's my site http://www.hibebefetaldoppler.com/fetal-doppler-questions-and-answers/ Thanks in advance
On-Page Optimization | | BistosAmerica0 -
Wordpress Pages vs. Posts
When building a blog to promote a particular affiliate offer, I usually like to use a static "page" homepage and then have my posts displayed on another part of my site. I've noticed that my wordpress pages almost always rank higher than wordpress posts and I can't explain it... Here are some possibilities I've thought of: XML Sitemap priority is set at 60% for pages and only 20% for posts My main navbar lists the pages which consequently means they get linked to on every page on my website. Some other phenomenon within the wordpress framework... If it helps, I use Thesis v1.8 on all my sites. I guess my ultimate question is: If pages do in fact rank higher than posts, is it worth it for me to go back and change the site structure on all my blogs which are using posts instead of pages. I know making major modifications like that can be disasterous but will it ultimately pay off? Thanks
On-Page Optimization | | drewhammond1