Access denied errors in webmaster tools
-
I notice today Ihave 2 access denied errors. I checked the help which says:
Googlebot couldn’t access a URL on your site because your site requires users to log in to view all or some of your content. (Tip: You can get around this by removing this requirement for user-agent Googlebot.)
Therefore I think it may be because I have added a login page for users and googlebot can't access it. I'm using wordpress and presume I need to amend the robots.txt to remove the requirement for google to log in but how do I do that?
Unless I'm misunderstanding the problem altogether!
-
Samuel
Mat's right - you need to speratae the "access denied" pages into two buckets;
-
those which you want indexed
-
those which you don't want indexed.
For those you want indexed, take them on a page by page basis to figure out why access is being denied. They might not all be because of password protection.
-Dan
-
-
Personally I wouldn't remove the requirement for googlebot if you want to protect that content. Once you do that content is public for two reasons:
1. Any nefarious user can instantly change their user agent and view your content
2. It will appear in the public search results
Much better would be to consider what is and isn't protected by a login. If you want it indexed then it should be seen as public and not behind a login. If it should be private, then protect it.
If you want it to appear in the search to attract people to the site in order to encourage them to register consider having cut down versions that are public and have a strong call-to-action on those to get people to register.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Backlink audit - anyone know of a good tool for manually checking backlinks?
Hi all, I'm looking at how to handle backlinks on a site, and am seeking a tool into which I can manually paste backlinks - is there a good backlink audit tool that offers this functionality? Please let me know! Thanks in advance, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Website structure - best tools to analyse and plan, visually
Hi - I am about to analyse and then re-plan the structure of a website and think it would be best to do it graphically - in the form of a chart. Are there any tools you would recommend to visualise the structure of an existing website (perhaps something that can scan and then represent a websites) - or plan out a new/revised website? Thanks in advance, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Crawl diagnostic how important is these 2 types of errors and what to do?
Hi,
Intermediate & Advanced SEO | | nicolaj1977
I am trying to SEO optimized my webpage dreamesatehuahin.com When I saw SEO Moz webpage crawl diagnostic I kind of got a big surprise due to the high no. of errors. I don’t know if this is the kind of errors that need to be taken very serious i my paticular case, When I am looking at the details I can see the errors are cause by the way my wordpress theme is put together. I don’t know how to resolve this. But If important I might hire a programmer. DUPLICATE ERRORS (40 ISSUES HIGH PRIORITY ACCORDING TO MOZ)
They are all the same as this one.
http://www.dreamestatehuahin.com/property-feature/restaurent/page/2/
is eaqual to this one
http://www.dreamestatehuahin.com/property-feature/restaurent/page/2/?view=list This one exsist
http://www.dreamestatehuahin.com/property-feature/car-park/
while a level down don’t exsit
http://www.dreamestatehuahin.com/property-feature/ DUPLICATE PAGE TITLE (806 ISSUES MEDIUM PRIORITY ACCORDING TO MOZ)
This is related to search results and pagination.
Etc. Title for each of these pages is the same
http://www.dreamestatehuahin.com/property-search/page/1 http://www.dreamestatehuahin.com/property-search/page/2 http://www.dreamestatehuahin.com/property-search/page/3 http://www.dreamestatehuahin.com/property-search/page/4 Title element is to long (405)
http://www.dreamestatehuahin.com/property-feature/fitness/?view=list
this is not what I consider real pages but maybe its actually is a page for google. The title from souce code is auto generated and in this case it not makes sense
<title>Fitness Archives - Dream Estate Hua Hin | Property For Sale And RentDream Estate Hua Hin | Property For Sale And Rent</title> I know at the moment there are properly more important things for our website like content, title, meta descriptions, intern and extern links and are looking into this and taking the whole optimization seriously. Have for instance just hired a content writer rewrite and create new content based on keywords research. I WOULD REALLY APPRICIATE SOME EXPERIENCE PEOPLE FEEDBACK ON HOW IMPORTANT IS IT THAT I FIX THIS ISSUES IF AT ALL POSSIBLE? best regards, Nicolaj1 -
Google Keyword Planner tool is not correct
Hi All, I know you are all know about Google keyword planner tool. As i know its shows most keywords searched totally wrong. One of the keyword searches less than (<10) but i got 20 exact keyword hits in only one business day and one of the keyword shows more then 10 K searches give us only 3-4 hits in one day.
Intermediate & Advanced SEO | | dotlineseo0 -
Duplicate Errors from Wordpress login redirects
I've some Duplicate issues showing up in Moz Analytics which are due to a Q&A plugin being used on a Wordpress website which prompts the user to login. There's a number of links looking like the one shown below, which lead to the login page: www.website.com/wp-login.php?redirect_to=http%3A%2F%2Fwww.website.com%question%2.... What's the best way to deal with this? -- extra info: this is only showing up in Moz Analytics. Google Webmaster Tools reports no duplicates.. I'm guessing this is maybe down to the 'redirect_to' parameter being effective in grouping the URLs for Googlebot. currently the wplogin and consequent redirects are 'noindex, follow' - I cannot see where this is being generated from in wp-login.php to change this to nofollow (if this will solve it).
Intermediate & Advanced SEO | | GregDixson0 -
Is the Tool Forcing Sites to Link Out?
Hi I have a tool that I wish to give to sites, it allows the user to get an accurate idea of their credit score with out giving away any personal data and with out having a credit search done on their file. Due to the way the tool works and to make the implementation on other peoples sites as simple as possible the tool remains hosted by me and a one line piece of Javascript code just needs to be added to the code of the site wishing to use the tool. This code includes a link to my site to call the information from my server to allow the tool to show and work on the other site. My questions are: Could this cause a problem with Google as far as their link quality goes? - Are we forcing people to give us a backlink to use the tool? (in the eyes of Google) or will Google not be able to read the Javascript / will ignore the link for SEO purposes? Should I make the link in the code Nofollow? If I should make the link a Nofollow any tips on how to make the most of the opportunity from a link building or SEO point of view? Thanks for your help
Intermediate & Advanced SEO | | MotoringSEO0 -
Matt Cutts Announces Disavow Google Webmasters Tool
Today at Pubcon Cutts announced this tool similar to Bing's - http://searchengineland.com/google-launches-disavow-links-tool-136826. My question is, has anybody used Bing's? Do you foresee any problems or issues to consider? Just checking before going ahead with using it 🙂 Thanks
Intermediate & Advanced SEO | | bradkrussell0 -
Strange affiliate links in web master tools
Hi All I have been looking in the Google webmasters tools account for a client and from out of nowhere they suddenly have 42,000 links last time I checked it was just over 4,000. It looks to me like some of our affiliates have links showing, they are formatted so it goes via a php script that looks like accomtracking.php?estid=1234&ref=123 this link appears 3 times in the source code of the page that is linking to my client but Google webmaster tools thinks it is linking 12,000 times. I assume this is a problem with the database where the link points to, issuing a unique link for each visit or something similar but I am not sure how to explain it to my client and how to correct it / stop it happening again. There are around 37,000 links like this to the site from just 6 referring sites and the client have had an unnatural link warning so I am anxious to get this fixed. Any help gratefully received. Sean
Intermediate & Advanced SEO | | SKE0