Googlebot stopped crawling
-
Hi All, One of my website stopped showing in SERP, after analysing in webmaster, found that Googlebot is not able to crawl. However it was working alright few days back. Try to investigate for panelisation, but no intimation found. I checked robot.txt for no follow etc but all seems to be ok. I resubmitted Sitemap in webmaster again, it crawled 250 pages out of 500 but it still site is not available in SERP (google), in bing it is ok.
Pl suggest the best possible solutions to try.
Thx
-
This might be a shot in the dark not knowing much about your site - can you check in Google Webmaster to see if you accidentally removed your website using the Remove URLs tool? I know of someone that accidentally did this when copying a pasting a URL, but accidentally only copied their main website address and not the full URL (oops!) and their site dropped out of Google SERPs rather quickly. Just a thought...
-
Very hard to say without more details. Does your site have unique, high quality content? If it's just duplicate content, Google may crawl it but won't necessarily show it in the SERPs.
Also, what does your backlink profile look like? Google allocates crawl budget based on your PageRank, so if Google isn't crawling all your pages, then you will want to acquire more external backlinks.
-
There may be many technical things going on with your robot.txt file, no-index tags, etc.
But where I would start first is with your website hosting company.
My guess - not having seen your site - is that you may be hosted on a site with a low-cost hosting provider. And you are experiencing downtime at random times that are affecting Google's ability to crawl your site.
The other clue that points me to your web hosting service is that Google tried to crawl 500 pages, but it was only able to handle 250.
What I would do is first look and see if your site is timing out under heavy loads / lots of visitors.
That's most likely the culprit.
I'd subscribe to a free site monitoring service that will ping the site every 5 minutes or so, and email you if it goes down.
Hope this helps!
-- Jeff
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl anamoly issue on Search Console
Has anyone checked the crwal anamoly issue under the index section on Search console? We recently move to a new site and I'm seeing a huge list of excluded urls which are classified as crawl anamoly (they all lead to 404 page). Does anyone know that if we need to 301 redirect all the links? Is there any other smarter/ more efficiently way to deal with them like set up canonical link (I thought that's what they're used for isn't it?) Thanks!
White Hat / Black Hat SEO | | greenshinenewenergy0 -
Google suddenly stops ranking a page for a "keyword" with same "keyword" in title tag. Low competition.
Hi all, We have released our next version of product called like "software 11", which have thousands of searches every month. So we have just added this same keyword "software 11" as page title suffix to one of the top ranking pages. Obviously this is the page has been added suddenly with "software 11" at page title, multiple header tags and 1 mention in paragraph. Google ranked it for 2 days and suddenly stopped showing this page in entire results for the same keyword we optimised the page for. Why does it happened? Does Google think that we are overdoing with this page and ignoring it? Thanks
White Hat / Black Hat SEO | | vtmoz0 -
Excluding Googlebot From AB Test - Acceptable Sample Size To Negate Cloaking Risk?
My company uses a proprietary AB testing platform. We are testing out an entirely new experience on our product pages, but it is not optimized for SEO. The testing framework will not show the challenger recipe to search bots. With that being said, to avoid any risks of cloaking, what is an acceptable sample size (or percentage) of traffic to funnel into this test?
White Hat / Black Hat SEO | | edmundsseo0 -
Does Google crawl and index dynamic pages?
I've linked a category page(static) to my homepage and linked a product page (dynamic page) to the category page. I tried to crawl my website using my homepage URL with the help of Screamingfrog while using Google Bot 2.1 as the user agent. Based on the results, it can crawl the product page which is a dynamic. Here's a sample product page which is a dynamic page(we're using product IDs instead of keyword-rich URLs for consistency):http://domain.com/AB1234567 Here's a sample category page: http://domain.com/city/area Here's my full question, does the spider result (from Screamingfrog) means Google will properly crawl and index the property pages though they are dynamic?
White Hat / Black Hat SEO | | esiow20130 -
My website is coming up under a proxy server "HideMyAss.com." How do I stop this from happening?
We've noticed that when we search our web copy in Google the first result is under a proxy server "HideMyAss.com," and our actual website is no where in sight. We've called Google and they really didn't have an answer for us (well the 2-3 people) we spoke with. Any suggestions or ideas would be greatly appreciated.
White Hat / Black Hat SEO | | AAC_Adam0 -
How to stop links from sites that have plagurized my blogs
I have been hit hard by Penguin 2.0. My webmaster explains that I have many links to my articles (a medical website with quality content) from "bad sites." These sites publish my articles with my name and link to my site and it appears I have posted my articles on their site although I have not posted them-theses sites have copied and pasted my articles. Is there a way to prevent sites from posting my content on their site with links to my site?
White Hat / Black Hat SEO | | wianno1681 -
Hits in H1 will improve ranking by regular crawling ?
Hello ! I was wondering if it's a good idea to keep the "Hits" in the H1 ? http://www.ibremarketing.com/item/netapp-e5400-storage-system.html Will Google come to check regularly the update (new information if I'm right) or if he will not like the idea to come back just for hits update. As I have very good results on this part of the website, I do not want to take any risk. Thanks a lot !
White Hat / Black Hat SEO | | AymanH0 -
Some pages of my website http://goo.gl/1vGZv stopped crawling in Google
hi , i have 5 years old website and some page of my website http://goo.gl/1vGZv stopped indexing in Google . I have asked Google webmaster to remove low quality link via disavow tool . What to do ?
White Hat / Black Hat SEO | | unitedworld0