Welcome to the Q&A Forum

Thanks, will give screaming frog a go. I own the tool but rarely seem to use it. Hopefully that finds out what is amiss.

Hi

Could i get a second opinion on the following please. ON a client site we seem to have had a massive drop off in google crawling in the past few weeks, this is linked with a drop in search impressions and a slight reduction in penalty. There are no warning messages in WMT to say the site is in trouble, and it shouldn't be, however cannot get to the bottom of what is going on.

In Feb the Kilobytes downloaded per day was between 2200 and about 3800, all good there. However in the past couple of weeks it has peaked at 62 and most days are not even over 3! Something odd has taken place.

For the same period, the Pages crawled per day has gone from 50 - 100 down to under 3.

At the same time the site speed hasn't changed - it is slow and has always been slow (have advised the client to change this but you know how it is....) Unfortunately I am unable to give the site url out so i understand that may impact on any advice people could offer.

Ive attached some screen shots from WMT below.

Many thanks for any assistance.

stats.png

I like to use both GA and a secondary stats tool to double check the figures.

Chris,

Thanks for the reply. Moments after posting (and after you replied) I discovered what it was. The site uses a bit of php code from an external site for tracking purposes. Something has broken at their end instead of my wordpress site displaying an error it is causing the 500 error and blocking the bots. I will remove the code and look for an alternative solution.

Thanks all.

Carl

Hi

Wondering if anyone could help out here please. I tried to crawl my website using screaming frog and got a 500 error warning (odd considering it worked yesterday). I tried google speed test and got the following message - PageSpeed Insights received a 500 response from the server. To analyze pages behind firewalls or that require authentication. However, it also crawled the site and gave me a score of 70/100. It showed the site thumbnail, as normal. This was updated this morning and it showed the latest one so must have crawled.

I can access the site, as can my 'people', the site is getting hits from real users in Google Analytics so must be working.

Am very puzzled how the site can be working and not be working at the same time! If possible i would rather not share the url. The site is a test of concept and is very messy looking. it is more a test to see what happens if i do x...

Many thanks

Carl

Hi all

Wondering if anyone could help me out here. I am seeing massive variations in WMT of google crawl stats on a site I run. Just wondering if this is normal (see attached image). The site is an eccommerce site and gets a handful of new products added every couple of weeks. The total no of products is about 220k so this is only a very small %.

I notice in WMT I have an amber warning under Server connectivity. About 10 days back I had warnings under DNS, Server and Robots. This was due to bad server performance. I have since moved to a new server and the other two warnings have gone back to green. I expect the Server connectivity one to update any day now.

Ive included the graph for this incase it is relevant here.

Many thanks for assistance.

Carl

crawlstats.png connect.png

Hi

Thanks for looking at the issue. There should be four working nameservers. I have four set in both WHM and at my domain registrar. I added two more two (3 and 4) so maybe they are taking a while to resolve around the web.

Will look at the SOA, thanks. Server and domain set up isn't at the top of my skill set. The domain you mention in this thread is just a testing domain to see what happens with a certain kind of content so it hasn't been treated too seriously to be honest.

thanks, very useful tool. None of my domains passed but i can still access them.  Will try moving away from A record set up and use nameservers instead

Hi

Anyone else having problems with Google's Pagespeed tool? I am trying to benchmark a couple of my sites but, according to Google, my sites are not loading. They will work when I run them through the test at one point but if I try again, say 15 mins later, they will present the following error message

An error has occured

DNS error while resolving DOMAIN. Check the spelling of the host, and ensure that the page is accessible from the public Internet. You may refresh to try again.

If the problem persists, please visit the PageSpeed Insights mailing list for support.

This isn't too much an issue for testing page speed but am concerned that if Google is getting this error on the speed test it will also get the error when trying to crawl and index the pages.

I can confirm the sites are up and running. I the sites are pointed at the server via A-records and haven't been changed for many weeks so cannot be a dns updating issue.  Am at a loss to explain.

Any advice would be most welcome. Thanks.

thanks for reminding me of the robots option. Overlooked that. Will propose it

Chris,

Thanks for the reply. Moments after posting (and after you replied) I discovered what it was. The site uses a bit of php code from an external site for tracking purposes. Something has broken at their end instead of my wordpress site displaying an error it is causing the 500 error and blocking the bots. I will remove the code and look for an alternative solution.

Thanks all.

Carl