Access denied in google webmaster tools
-
Hi I have just checked on my google webmaster tools and it is showing i 11 urls that are coming back as access denied. Now the urls are working, and they have been redirected using 301 redirect, so i have done everything right but for some reason google is not able to crawl them.
Does anyone know what i have done wrong for it to come back as access denied and how i can solve this problem. the site is www.in2town.co.uk many thanks
| | | |
| | 2 | Gardening/Gardening-Advice-What-is-Hydroponic-Gardening/menu-id-4991 | 403 | 4/11/13 |
| | 3 | Top-Showbiz-News/Super-Injunctions-Are-Right-Says-Hugh-Grant | 403 | 4/29/13 |
| | 4 | Entertainment-Tonight/Cheryl-Cole-wants-to-spice-up-The-X-Factor | 403 | 4/11/13 |
| | 5 | Tiger-Woods-paid-10000-a-time-for-sex | 403 | 4/20/13 |
| | 6 | Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/menu-id-4448 | 403 | 4/24/13 |
| | 7 | News-Showbiz/Doctor-Who-changed-my-life-says-Matt-Smith | 403 | 4/29/13 |
| | 8 | The-Latest-Health-News/Hypnosis-Hypnotherapy-for-Relationships/menu-id-4744 | 403 | 4/11/13 |
| | 9 | Soap-Gossip-Latest-News/Emmerdale-Marks-bit-on-the-side-comes-to-home-farm/menu-id-4615 | 403 | 4/11/13 |
| | 10 | news/eastenders/ | 403 | 4/11/13 |
| | 11 | entertainment-news/Prince-William-Stag-Do-To-Be-Held-in-Cape-Town | 403 | 3/24/13 || | |
-
cheers for this, i have contacted the company to see what they say about this issue and hopefully it will be resolved.
-
As I said the clue is likely in the message you get from the front end "Forbidden Access (flooding)"
If you search for this, the results all seem to mention joomla and that module. If you look through those results there are some mentions of the security features of this SEF module and how to turn them on/off. It is impossible to say if this is 100% the cause of your issue, but if your hosting company say everything is fine, and the message shown is specific to this joomla module, then it is a likely candidate. All things being equal, try turning off this security feature and see if the access denied errors in GWT go away.
-
just going into my webmaster tools and it says the following
| | Response Code | Detected |
| --- | --- | --- |<colgroup><col style="width: 45px;"><col style="width: 80px;"><col><col style="width: 120px;"><col style="width: 90px;"></colgroup>
| | 1 | Headlines-Celebrity-/Simon-Cowell-The-Wedding-Is-Back-On | 403 | 4/20/13 |
| | 2 | Gardening/Gardening-Advice-What-is-Hydroponic-Gardening/menu-id-4991 | 403 | 4/11/13 |
| | 3 | Top-Showbiz-News/Super-Injunctions-Are-Right-Says-Hugh-Grant | 403 | 4/29/13 |
| | 4 | Entertainment-Tonight/Cheryl-Cole-wants-to-spice-up-The-X-Factor | 403 | 4/11/13 |
| | 5 | Tiger-Woods-paid-10000-a-time-for-sex | 403 | 4/20/13 |
| | 6 | Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/menu-id-4448 | 403 | 4/24/13 |
| | 7 | News-Showbiz/Doctor-Who-changed-my-life-says-Matt-Smith | 403 | 4/29/13 |
| | 8 | The-Latest-Health-News/Hypnosis-Hypnotherapy-for-Relationships/menu-id-4744 | 403 | 4/11/13 |
| | 9 | news/eastenders/ | 403 | 4/11/13 |
| | 10 | entertainment-news/Prince-William-Stag-Do-To-Be-Held-in-Cape-Town | 403 | 3/24/13 ||
when i have looked at more info on this it says the following
Access denied errors
In general, Google discovers content by following links from one page to another. To crawl a page, Googlebot must be able to access it. If you’re seeing unexpected Access Denied errors, it may be for the following reasons:
- Googlebot couldn’t access a URL on your site because your site requires users to log in to view all or some of your content. (Tip: You can get around this by removing this requirement for user-agent Googlebot.)
- Your robots.txt file is blocking Google from accessing your whole site or individual URLs or directories.
- Test that your robots.txt is working as expected. The Test robots.txt tool lets you see exactly how Googlebot will interpret the contents of your robots.txt file. The Google user-agent is Googlebot. (How to verify that a user-agent really is Googlebot.)
- The Fetch as Google tool helps you understand exactly how your site appears to Googlebot. This can be very useful when troubleshooting problems with your site's content or discoverability in search results.
- Your server requires users to authenticate using a proxy, or your hosting provider may be blocking Google from accessing your site.
i have asked my hosting company about this and they say everything is fine.
any help would be great to solve this
| |
-
going to go through these today as they may have changed since the update. so you feel the sh404sef could be causing the blocking problems, i will contact them.
-
Hi Tim,
Well both those urls you give for the 301 are returning a 404, but I don't think they are the cause of your original problem which is the access denied issue. For that I am pretty sure you need to be looking at that joomla SH404SEF module.
-
hi both sides are there, for example from above,
Redirect 301 /Lingerie-Brands-expert-says-Lingerie-improves-your-sex-life /lingerie-helps-improve-your-sex-life
so i have the original page and then pointing to the destination page
i just do not understand after all the checks i have done why the error is happening
-
Hi Tim,
Your robots.txt looks ok from what I can tell. The 301s dont look odd (although what you have there is only one side of them right? I don't see the final page).
I think the clue is the message you get on the front end "Forbidden Access (flooding)". If you search for this phrase you start seeing references to the joomla SH404SEF module. See here for example: http://forum.joomla.org/viewtopic.php?p=1368937
I am not a joomla expert, but maybe it is a joomla issue instead of a server one, worth looking into.
-
they have also said my 301 redirects are causing the problems but i thought i had done this correctly
here are some of my redirects
The lines that are causing the issue are:
Redirect 301 /Jennifer-Aniston-upset-over-Brad-Pitt-Marriage /news/have-your-say/jennifer-aniston-upset-over-brad-pitt-marriage
Redirect 301 /In2town-Gossip/Liz-Hurley-Wants-Her-Husband-Back /news/have-your-say/liz-hurley-wants-her-husband-back
Redirect 301 /News-Celebrity/Take-That-and-Robbie-Williams-do-it-again /news/have-your-say/take-that-and-robbie-williams-do-it-again
Redirect 301 /Kevin-McCloud-does-not-like-the-word-Poverty /news/have-your-say/kevin-mccloud-does-not-like-the-word-poverty
Redirect 301 /Latest-Travel-News/Singapore-Tourist-Information-Singapore-a-must-for-Holidays/menu-id-4592 /news/holidays/singapore-tourist-information-singapore-a-must-for-holidays
Redirect 301 /Travel-Articles/Holiday-makers-are-rushing-to-buy-cheap-flights-to-Benidorm/menu-id-4998 /news/flight-news/holiday-makers-are-rushing-to-buy-cheap-flights-to-benidorm
Redirect 301 /The-Latest-Health-News/Stop-Biting-Your-Nails/menu-id-4744 /news/healthy-living/stop-biting-your-nails-with-hypnotherapy
Redirect 301 /Woman-celebrates-after-losing-weight-with-Weight-Loss-Hypnosis /news/gastric-band-hypnotherapy/woman-celebrates-after-losing-weight-with-gastric-band-hypnosis
Redirect 301 /Health-News-/-Stop-Smoking-Hypnosis-really-works-says-expert /news/health/stop-smoking-hypnosis-really-works-says-stop-smoking-expert
Redirect 301 /Animal-Health-News/Pet-Advice-Kennel-Cough-Advice-for-your-Pets/menu-id-4954 /news/dog-care/dog-kennel-cough
Redirect 301 /Travel-News/Flying-to-Australia-consumers-are-turning-their-back-on-Travel-Agents-over-cheap-flights/menu-id-4592 /news/flight-news/flying-to-australia-consumers-turn-their-backs-on-travel-agents-for-cheap-flights-to-australia
Redirect 301 /The-Latest-Health-News/Childbirth-Hypnotherapy/menu-id-4744 /news/health/childbirth-hypnotherapy
Redirect 301 /Travel-News/Brazil-Holidays-is-becoming-a-huge-hit-with-British-Tourist/menu-id-4592 /news/holidays/brazil-holidays-is-becoming-a-huge-hit-with-british-tourist
Redirect 301 /Celebrity-Gossip-Celebrity-News-and-latest-celebrity-gossip /Showbiz-Gossip
Redirect 301 /Latest-Travel-News/Travel-Magazine-reveals-secrets-to-finding-Cheap-Flights /news/flight-news/saving-money-on-flights
Redirect 301 /Lingerie-Brands-expert-says-Lingerie-improves-your-sex-life /lingerie-helps-improve-your-sex-life -
contacted my hosting company and they said the following robots file could be blocking google and causing the 403 errors, i thought it looked standard to me, can anyone please have a look and let me know if my robots file is causing the problem
many thanks
If the Joomla site is installed within a folder such as at
e.g. www.example.com/joomla/ the robots.txt file MUST be
moved to the site root at e.g. www.example.com/robots.txt
AND the joomla folder name MUST be prefixed to the disallowed
path, e.g. the Disallow rule for the /administrator/ folder
MUST be changed to read Disallow: /joomla/administrator/
For more information about the robots.txt standard, see:
http://www.robotstxt.org/orig.html
For syntax checking, see:
http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/ -
thank you for this, what would be the best way to solve this, as this must be affecting my rankings
-
Hi Tim,
It looks like you have some sort of server setup that is trying to block dos attacks or similar. If you put your site into screaming frog after the first few pages it starts returning 403 errors (access denied). If you then look at a page in a browser your see the message: Forbidden Access (flooding).
Is it likely that this is happening when google is trying to spider the site also?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google + and Google Knoladge Graph
I am trying to get things to match up for the company brand websearch and the Google + page and we have had it for years now The knowledge graph on Google is showing the map, address and name (shown in attached image), but is not linked to a G+ page, as when i click the "Are you the business owner?" its is trying to make me create a new G+ business page. Anyone have any ideas on this? Also does the wiki name have to be exact for it to show? As for phone number would that be coming from the DNS record as that is nowhere in the markup rich snippet or normal markup Thanks in advance LC9cWdG
Technical SEO | | David-McGawn0 -
Google Bot Noindex
If a site has the tag, can it still be flagged for duplicate content?
Technical SEO | | MayflyInternet0 -
Verify all versions of site in Bing Webmaster Tools
Hello, We recently migrated our site to a new shopping cart, https, and from www to non-www, and it's been a rough transition. We've lost a lost of traffic particularly in Bing. All the versions of our site are verified Google WMT, sitemaps are submitted correctly, etc. Unfortunately, this was not done for Bing. Currently only the new version of our site (https, non-www) is verified in Bing WMT. Do we have to verify all versions of our site in Bing, the way they are in Google WMT? Also, now that it's been a few months since the switch, should we still submit a site move to Bing WMT or is it too late? Thanks in advance!
Technical SEO | | whiteonlySEO0 -
Google not indexing my website
Hi guys, We have this website http://www.m-health-expo.nl/ but it is not indexed by google. In webmaster tools google says that it can not fetch the site due to the robots.txt but i do not see any faults in it. http://www.m-health-expo.nl/robots.txt Do you see something strange, it really bothers me.
Technical SEO | | RuudHeijnen0 -
Updating Meta Tag error quickly besides submit to index in Webmaster Tools
For a conference page marketing built the meta tag didn't have correct year and date of the conference. I updated and used webmaster tools submit to index to try and get it updated in google search quickly but meta tag has not updated. Are there other avenues to get this corrected?
Technical SEO | | inhouseninja0 -
Crawling issues in google
Hi everyone, I think i have crawling issues with one of my sites. It has vanished form Google rankings it used to rank for all services i offered now it doesn't anymore ever since September 29th. I have resubmitted to Google 2 times and they came back with the same answer: " We reviewed your site and found no manual actions by the web spam team that might affect your site's ranking in Google. There's no need to file a reconsideration request for your site, because any ranking issues you may be experiencing are not related to a manual action taken by the webspam team. Of course, there may be other issues with your site that affect your site's ranking. Google's computers determine the order of our search results using a series of formulas known as algorithms. We make hundreds of changes to our search algorithms each year, and we employ more than 200 different signals when ranking pages. As our algorithms change and as the web (including your site) changes, some fluctuation in ranking can happen as we make updates to present the best results to our users. If you've experienced a change in ranking which you suspect may be more than a simple algorithm change, there are other things you may want to investigate as possible causes, such as a major change to your site's content, content management system, or server architecture. For example, a site may not rank well if your server stops serving pages to Googlebot, or if you've changed the URLs for a large portion of your site's pages. This article has a list of other potential reasons your site may not be doing well in search. " How i detected that it may be a crawling issue is that 2 weeks ago i changed metas - metas are very slow in getting updated and for some of my pages never did update Do you know any good tools to check for bad code that could slow down the crawling. I really don't know where to look other than issues for crawling. I validated the website with w3c validator and ran xenu and cleaned these up but my website is still down. Any ideas are appreciated.
Technical SEO | | CMTM0 -
Is it OK for a sitemap to appear as a "Top URL" in Google Webmaster?
I'm using Google Webmaster (alongside other tools) to understand how Google is indexing my site. One of the tools is "Content Keywords", where it lists keywords that Google sees as significant for your site. The keywords shown are generally fine, but when I click on an individual word, I am often seeing our sitemap as one of the "Top URLs" that the keyword is found on (our sitemap is at system/sitemap1.xml.gz) - is this OK? Obviously I don't want to add the sitemap URL to robots.txt, but I also want to ensure that 'real' user-focused pages (e.g. our homepage) appear higher in the "Top URLs" list for the keywords, as I'm assuming this is an indicator of how the site is performing in search. Any help appreciated!
Technical SEO | | anilababla0 -
Google Quality Algorithm Update
I'm curious what correlations or impacting variables SEO professionals have found that have increased or decreased ranking with the most recent algorithm change. It appears that many innocent sites have fallen victim, especially larger sites. It also appears that Google is maintaining that specific sites were not targeted... Meaning there must be proven characteristics.
Technical SEO | | douglaskarr0
