Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: / -
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO audit on a beta site
HI there, Is there much point conducting an SEO site audit on a site that has not yet launched and is protected behind a login? Presumably none of the usual SEO tools (Moz, Screaming Frog etc) can crawl this site becuase it is all locked behind a login. Would it be better to launch it and then do a site audit? Thanks
On-Page Optimization | | CosiCrawley0 -
To change or not to change site URL structure?
I am learning my way around SEO, having always used professionals for it in the past on previous businesses i have decided to do it myself and learn more about it. Now the dilemma i am up against is i recently changed some of my permalinks on quite a few main pages throughout the site. The site launched in April this year so we're quite new. The problem is since my last change i have not seen any increase...a decrease which in fact hasn't recovered at all. Having now analysed them in more depth and read up more on the whole subject of SEO, (which is endless) i have put together a complete new strategy; with this increased understanding of what i am doing (but by no means conclusive) i want to complete a full overhaul on all SEO (via Wordpress which i use along with YOAST SEO tools), ensuring i have all my keywords, permalinks and descriptions spot on throughout every page, post and picture. I spent a lot of time mapping these out, ensuring there is no Focus keyword duplication, and that the site is relatively flat in terms of its layout. What i am unsure about now is whether changing my permalinks again is a bad thing to do?
On-Page Optimization | | MrCostello
Could it permanently damage my rep going forward?
Should i just focus on my content and keywords/descriptions? I am at a loss as i don't want to do irreparable damage to our reputation. The site is still reasonable easy to manage so changing now is the best time to do it, but if changing the URLs is a waste of time then i may just forget that and just work on the keywords, descriptions and content. Advice is 'oh so welcome' 🙂0 -
Robots file include sitemap
Hello, I see that google, facebook and moz... have robots.txt include sitemap at the footer.
On-Page Optimization | | JohnHuynh
Eg: http://www.google.com.vn/robots.txt Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this? Thanks,0 -
Duplicate content issue
Hello, I got duplicate content issue on my home page : examplesite.com
On-Page Optimization | | digitalkiddie
examplesite.com/index.html Those page urls are with duplicate content. If in index.html i use 301 redirect like that : Header( "HTTP/1.1 301 Moved Permanently" );
Header( "Location: http://examplesite.com" );
?> would i loose any page authority ? sorry for the newbie question0 -
Recommended Wordpress themes for great SEO
Hi there, I'm trying to decide which theme to use. My main concern is if the theme has been built using a stable framework to ensure plugin compatibility for SEO. Also a general question, does the homepage need to have a certain amount of text content for SEO, or can other pages be SEO optimised with text / keyword? Some of the nicer themes seem to have a sparce text to image ratio on the homepage. I quite like this as an initial 'welcome' but worry this isn't great for SEO. Any suggested themes / advice will be welcome. Thanks
On-Page Optimization | | well-its-1-louder0 -
Checking for content originality in a site
two part question on original content How would you go about checking if a site holds original content accept the long search quary within Google? ans also if I find many sites carrying my content and I am the original source should I replace the content? thanks
On-Page Optimization | | ciznerguy0 -
Issue: Rel Canonical
My SEO Report shows issues: Rel Canonical I have a wordpress website each page has its content but I'm getting errors from my SEOMOZ report. I instaledl the yoast plug in to fix the issue but I'm still getting 29 errors. Wordpress 3.4.1
On-Page Optimization | | mobiledudes0 -
How deep should I go with a directory site?
I am creating a new site which has a directory component. Based on what I have ready I am inclined to keep the site architecture as flat as possible. However, the natural layout that I have come up with in my head has the directory listings 5 or 6 pages deep in the site structure. I saw in another post that someone in a similar situation was suggesting that going deep like this is fine so long as there are many internal links to the deeper pages to indicate that they are important. Should I make a conscious effort to make the site architecture as flat as possible? Are there any specific guides/resources that address this particular issue that I should be aware of? Thanks!
On-Page Optimization | | fastestmanalive0
