Bots gone from site for 1-2 hours
-
Hi everyone,
From the beginning of March bots and crawlers begin to disappear from site completely for 1-2 hours and then come back again. It never happened earlier, they constantly crawled our site without stopping. We didn't make any changes on the site.
After that pages and images start to disappear from G index.
Does anyone have the same trouble?
-
Assuming you are talking about the website in your profile...here is what I would check: Do you have any content duplication between your multiple language subdomains ? Do you have XML Sitemaps for your pages ? Are they submitted via the Webmaster Console ? If yes, what %age of your pages do you see indexed vs how many you submitted ? Have you considered doing an Images XML Sitemap ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawling with Firewall Plugin
Just wondering if anyone has any experience with the WordPress Simple Firewall plugin. I have a client who is concerned about security as they've had issues in that realm in the past and they've since installed this plugin: https://wordpress.org/support/view/plugin-reviews/wp-simple-firewall?filter=4 Problem is, even with a proper robots file and appropriate settings within the firewall, I still cannot crawl the site with site crawler tools. Google seems to be accessing the site fine, but I still wonder if it is in anyway potentially hindering search spiders.
Technical SEO | | BrandishJay0 -
Is a micro site the way to go?
Hello, a client has asked us today to quote for how much it would cost them to get a micro site built. A Google employee has told them that because their current URL doesn't include .co.uk or.com it is simply: brandname.word that it will be harder for them to get their website to rank. My understanding is that micro sites aren't a good solution for any problem as Google doesn't like them. Would it be better for them to buy a .co.uk (they are a UK company) url and then redirect the url to their current website or is there a better solution? Many thanks
Technical SEO | | mblsolutions0 -
Then why my site is not ranking
My website's DA and PAs are good compare with my competitors. Then why my site is not ranking.
Technical SEO | | Somanathan0 -
301 redirect from sites closing down
Hi We have around 10 supplementary sites that have links to our site which are now closing down but are out of our control. We could have access to their domains so how could we maintain the link juice from these old sites which are going to our new site? However there will be no websites left on these old supplementary just domain names
Technical SEO | | ocelot0 -
Google Shows 24K Links b/w 2 sites that are not linked
Good Morning, Does anyone have any idea why Google WMT shows me that i have 24,101 backlinks from one of my sites ( http://goo.gl/Jb4ng ) pointing to my other site ( http://goo.gl/JgK1e ) ... These sites have zero links between them, as far as I can see/tell. Can someone please help me figure out why Google is showing 24k backlinks? Thanks
Technical SEO | | Prime850 -
Google Custom Site Search
I am an admin on a google custom site search account. I am also the owner of a verified webmaster tools account for the same site. The Custom Search control panel will not let me add URL's or a Site map for on demand indexing, but says "you must submit a sitemap of your own verified sites". Has anyone else has this issue? Does the Owner of the custom search account have to be the owner of the webmaster account, or can the logged in admin be? Thanks
Technical SEO | | SEMPassion0 -
/$1 URL Showing Up
Whenever I crawl my site with any kind of bot or a sitemap generator over my site. it comes up with /$1 version of my URLs. For example: It gives me hdiconference.com & hdiconference.com/$1 and hdiconference.com/purchases & hdiconference.com/purchases/$1 Then I get warnings saying that it's duplicate content. Here's the problem: I can't find these /$1 URLs anywhere. Even when I type them in, I get a 404 error. I don't know what they are, where they came from, and I can't find them when I scour my code. So, I'm trying to figure out where the crawlers are picking this up. Where are these things? If sitemap generators and other site crawlers are seeing them, I have to assume that Googlebot is seeing them as well. Any help? My developers are at a loss as well.
Technical SEO | | HDI0 -
How do I set up a site review for a password protected site?
We need to conduct a SEO analysis for a website that is on a private, password protected development site -- is there anyway for SEOMoz tools to access and analyze a PW protected site? Thank you, Sara Merten
Technical SEO | | kev110