Access denied errors in webmaster tools
-
I notice today Ihave 2 access denied errors. I checked the help which says:
Googlebot couldn’t access a URL on your site because your site requires users to log in to view all or some of your content. (Tip: You can get around this by removing this requirement for user-agent Googlebot.)
Therefore I think it may be because I have added a login page for users and googlebot can't access it. I'm using wordpress and presume I need to amend the robots.txt to remove the requirement for google to log in but how do I do that?
Unless I'm misunderstanding the problem altogether!
-
Samuel
Mat's right - you need to speratae the "access denied" pages into two buckets;
-
those which you want indexed
-
those which you don't want indexed.
For those you want indexed, take them on a page by page basis to figure out why access is being denied. They might not all be because of password protection.
-Dan
-
-
Personally I wouldn't remove the requirement for googlebot if you want to protect that content. Once you do that content is public for two reasons:
1. Any nefarious user can instantly change their user agent and view your content
2. It will appear in the public search results
Much better would be to consider what is and isn't protected by a login. If you want it indexed then it should be seen as public and not behind a login. If it should be private, then protect it.
If you want it to appear in the search to attract people to the site in order to encourage them to register consider having cut down versions that are public and have a strong call-to-action on those to get people to register.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Handle a Soft 404 error to an admin page in WordPress
I'm seeing this error on Google Webmaster Console: | URL: | http://www.awlwildlife.com/wp-admin/admin-ajax.php | | | Error details | Linked from | | |
Intermediate & Advanced SEO | | aj613
| Last crawled: 11/15/16First detected: 11/15/16 The target URL doesn't exist, but your server is not returning a 404 (file not found) error. Learn more Your server returns a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404). This creates a poor experience for searchers and search engines. More information about "soft 404" errors | Any ideas what I should do about it? Thanks!0 -
Webmaster tools: which one do you use? Yandex Yay or Nay?
I usually verify websites on Google and Bing Webmaster. How important it is to verify on Yandex Webmaster if Russia is not one of the targeted locations?
Intermediate & Advanced SEO | | selectitaly0 -
Using "Read More" buttons as a tool to cram in Content
Hi Mozzers! Let's say our website is clean, professional, and minimalistic. Can we use a "read more" button that will expand the text on the page to increase the amount of content while (unless clicked) not impacting the appearance? I want to make sure I am not violating Google Webmaster's guidelines for "Hidden Text" Thanks!
Intermediate & Advanced SEO | | Travis-W0 -
How To Remve Rel Canonical Error from site
Hello friends, I have a site there I install all in one SEO plugin when I add my site at seomoz.org after the crawling results it so there are a penalty of Rel Conanical tag error but when I see my editor code there I see that all in one seo automatically giving rel conanical tag. Now I don’t understand that why seomoz giving these errors. Please help me to resolve this problem.
Intermediate & Advanced SEO | | KLLC0 -
SEO and Pictures tool
Hello, I need to share pictures albums. I would like to know if any of you have an opinion on the best tools available to share pictures on the web? When I say 'the best tool' I mean from an SEO perspective. So, based on your experience, is there tools with which I have better chances to get my pictures indexed? Thanks !! Note: CNET has created a great article that present the major players
Intermediate & Advanced SEO | | EnigmaSolution0 -
Do I need a canonical tag on the 404 error page?
Per definition, a 404 is displayed for different url (any not existing url ...). As I try to clean my website following SEOmoz pro advices, SEOmoz notify me of duplicate content on urls leading to a 404 🙂 This is I guess not that important, but just curious: should we add a cononical tag to the template returning the 404, with a canonical url such as www.mysite.com/404 ?
Intermediate & Advanced SEO | | nuxeo0 -
Can I Improve Organic Ranking by Restrict Website Access to Specific IP Address or Geo Location?
I am targeting my website in US so need to get high organic ranking in US web search. One of my competitor is restricting website access to specific IP address or Geo location. I have checked multiple categories to know more. What's going on with this restriction and why they make it happen? One of SEO forum is also restricting website access to specific location. I can understand that, it may help them to stop thread spamming with unnecessary Sign Up or Q & A. But, why Lamps Plus have set this? Is there any specific reason? Can I improve my organic ranking? Restriction may help me to save and maintain user statistic in terms of bounce rate, average page views per visit, etc...
Intermediate & Advanced SEO | | CommercePundit1 -
Shall I fix "most Common Errors" for a website that ranked top 3 on Google (difficult KW)?
How can SEOmoz "most Common Errors*" under "Crawl Diagnostics" advice can be right for a good site organic? Site is well ranked top 3 on Google (difficult KW). If I go ahead and fix these errors, I might hurt my SEO , no? like: Too Many On-Page Links 302 (Temporary Redirect) Title Element Too Long (> 70 Characters) Missing Meta Description Tag
Intermediate & Advanced SEO | | Elchanan0