Google haveing problems accessing part of my site
-
hi my site is, www.in2town.co.uk and for a few weeks now google has had trouble accessing part of my site.
Today googlewebmaster tools tells me that google is having major problems it shows, 123 pages where access were denied.
i have spoken to my hosting company who could not find a problem, so not sure what to do now. can anyone please give me advice on what the problem may be.
any help would be great
-
That's correct. I don't know the specifics, however in this case, I would assume the reduced frequency/volume would mean less trouble dealing with the hosting provider's internal challenges.
-
i have read it time and time again and do not understand how it can help. if you reduce the crawl rate then that means you reduce the number of pages and the number of times google visits your site or am i wrong
-
Fascinating how some problems can come up like this - it's not anything I'd ever seen - curious to see whether reducing Google's crawl rate will help.
-
i have asked about this problem on the google webmaster forum
http://productforums.google.com/forum/#!mydiscussions/webmasters/BogC5OZqdyM
and people are saying that it is a crawler issue with my hosting company,
they are suggesting a number of things that could be wrong which includes
Many hosting providers need to have mechanisms to throttle/limit either the number of requests or bandwidth to their customers sites.
However in this instance, they are not allowing for search engine crawlers which typically make numerous requests in quick succession. Significantly, their server's response is technically incorrect (RFC 2616), it should either not respond, or respond with a 503 (service unavailable). Google is aware of such incorrect 403s and will eventually try to crawl them again.To mitigate the issue somewhat, you can reduce your site's crawl rate in Google's Webmaster Tools
-
This confirms the need to get an expert involved - one experienced in these types of issues - a programmer or systems administrator type expert, not necessarily an SEO expert.
-
the site has been built from a template but this problem has only started to happen in the past month and the site is a couple of years old
-
unfortunately I don't know exactly what else to suggest. Who built the site? Is it a custom site, or was it automatically built using a hosting company automated site builder?
If it's a custom site or you have a developer you've worked with to build it, speak with them. if it's a site automatically built using a hosting site building system, only they can help resolve it. If not, you'll need to get a different site built. The problem you are describing should not be happening.
-
hi, when i put in http://www.in2town.co.uk on the tool you told me about it brings up some pages that are fine and some that are 403, and the 14th one down is the url as above. that comes up with a 403 however, when i just put the url on its own, it comes up as a 200, which is strange.
Have you got any ideas on what i should say to the hosting company. i have spoken to them a number of times and all i get is, everything is fine but it is not
-
When I put that URL in, I get a "200" "OK" response. If you get a "403 forbidden, you need to speak with the site systems (server) administrator to find out if they can pinpoint the problem. Something is broken on the site at the server level to cause that. Even if it's only intermittently broken.
-
hi, thanks for this. i have used that tool and it has come up with as an example the following
http://www.in2town.co.uk/news/parenting/parents-need-to-act-against-child-obesity 403 forbidden
i am not sure why this has happened and this is the same result as i am being shown in google webmaster tools.
-
One step I would suggest is that I would suggest using a tool like Screaming Frog. It can crawl your site and report on problems or broken internal links on the site.
I would also go into Google Analytics and go into the Content/Speed/Page Speed data. I would then expand the date range to cover from before they started listing problems until the day before you run the report (the current day can lag in updating in Google Analytics).
From there, look to see if there are any specific days where Google shows the page speed average (in the timeline chart) as being unusually high. There may be days that are more problematic than others.
Finally, with that same date range in place, I would filter down the list of pages in the speed report down to show me one of the pages Google lists as being unreachable. Then do the same with some others. See if Google shows any odd page speed times on those.
I would also then go to URIValet.com and run a check on some of those pages as well to see if URIValet shows the pages reachable and returning a proper "200" "okay" header message.
This is not a guarantee fix, however it can help bring answers.
-
the site is www.in2town.co.uk www.in2town.co.uk can you try it with the link and without the link, not sure why you cannot get on the site but will be interested to find out.
this problem of google have access errors has been going on for some weeks now and i do not understand what is going on
-
Hi Claire,
I was able to view your robots.txt file but nothing else. I can't seem to get to the site. Is the URL correct?
Dana
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How ask Google to de index scrapper sites?
While doing text Google searches for various keywords I have found two sites that have scrapped pages from my site which goes by an old URL of www.tpxcnex.com and a new URL of www.tpxonline.com www.folder.com is one of the sites and if you try to visit that site or any of the scrapped Google index listing, Chrome warns you not to. How can I ask Chrome to deindex www.folder.com or another scrapper site, or atleast deindex the URLs which have clearly scrapped my content?
Technical SEO | | DougHartline0 -
Site Audit Tools Not Picking Up Content Nor Does Google Cache
Hi Guys, Got a site I am working with on the Wix platform. However site audit tools such as Screaming Frog, Ryte and even Moz's onpage crawler show the pages having no content, despite them having 200 words+. Fetching the site as Google clearly shows the rendered page with content, however when I look at the Google cached pages, they also show just blank pages. I have had issues with nofollow, noindex on here, but it shows the meta tags correct, just 0 content. What would you look to diagnose? I am guessing some rogue JS but why wasn't this picked up on the "fetch as Google".
Technical SEO | | nezona0 -
Why is my site crawled so much more in Bing then in Google?
I recently setup Cloudflare so I can see how much my site is being crawled. It looks like Bing is crawling me about 3 times as much as Google. Any ideas on why that would be or what I should check?
Technical SEO | | EcommerceSite0 -
Site being indexed by Google before it has launched
We are currently coming towards the end of migrating one of our retail sites over to magento. To our horror, we find out today that some pages are already being indexed by Google, and we have started receiving orders through new site. Do you have any suggestions for what may have caused this? Or similarly, what the best solution would be to de-index ourselves? We most recently excluded anything with a certain parameter from robots.txt - could this being implemented incorrectly have caused this issue? Thanks
Technical SEO | | Sayers0 -
Look of google results
Can anyone tell me why some google results show the main page and then a listing of all subsequent pages (i.e. results for SEOMOZ) while others just show the main page with nothing under it. I have two different sites (one personal the other biz) and they both show their search results differently. Is it something in the site creation or how it is crawled by google? Thanks. bKs3C
Technical SEO | | STF0 -
About Google Spider
Hello, people! I have some questions regarding on Google spider. Many people are saying that "Google spiders only have US IP address." Is this really true? But I also saw video from Google's offical blog and it said "Google spider come from all around the world." At this point I am really confused. Q1) I researched and it seems like Google spiders have only US IP address. THen what does exactly mean by "Google spider come from all around the world."? Q2) If Google spider have only US IP address, what happen to site which use IP delivery? Is this means that Google spider always redirect to us site since they only have US IP? Can anyone help me to understand?? One more questions! When Google analyzing for cloaking issue, do you think Google analyze when spider crawls the site or after they crawled the site?
Technical SEO | | Artience0 -
Access To Client's Google Webmaster Tools
Hi, What's the best/easiest way for a client to grant access to his Google Webmaster Tools to me? Thanks! Best...Michael
Technical SEO | | 945010 -
Should I 301 my non-www accesses to www accesses?
We have external links pointing to both mydomain.com and www.mydomain.com. I read this: http://www.stepforth.com/resources/web-marketing-knowledgebase/non-www-redirect/ and wondered if I should add this to my .htaccess file: RewriteCond %{HTTP_HOST} ^mydomain.com
Technical SEO | | scanlin
RewriteRule (.*) http://www.mydomain.com/$1 [R=301,L] so that the link juice all flows to the www version of the site? Any reason not to do it?0