Bots gone from site for 1-2 hours
-
Hi everyone,
From the beginning of March bots and crawlers begin to disappear from site completely for 1-2 hours and then come back again. It never happened earlier, they constantly crawled our site without stopping. We didn't make any changes on the site.
After that pages and images start to disappear from G index.
Does anyone have the same trouble?
-
Assuming you are talking about the website in your profile...here is what I would check: Do you have any content duplication between your multiple language subdomains ? Do you have XML Sitemaps for your pages ? Are they submitted via the Webmaster Console ? If yes, what %age of your pages do you see indexed vs how many you submitted ? Have you considered doing an Images XML Sitemap ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can you have 2 different websites on 1 webmaster tools account
Someone set up both our sites on the one webmaster tools account is this the best way to do it or should we have 2 different accounts. We are having problems with our site verification not working and our google shopping feeds not working could this be the cause.
Technical SEO | | CostumeD0 -
Traffic Drop in 1 month!
Hi guys, ok, i'm at the end of my line with this one, my site www.golfclothingdirect.co.uk has taken a pounding since 4th April and i'm blowed if i can find a real reason for it. Traffic has been increasing slowly in line with budget, sales were picking up, and coming into season should have been better - 24th & 25th robot had an access issue - host contacted and said there had been a site issue, but resolved and now all A OK. Sales are terrible and traffic is grinding. All comments greatly taken on board! Thanks Mark
Technical SEO | | UrbanMark0 -
How Often is Site Crawled
Good morning- I saw some errors in my first crawl and immediately removed the pages from my website. I then re-created my XML sitemap and uploaded to Google. The question I have is will the site be crawled to recognize the changes in the next day or so? The pages were just placed on the site as test pages and never removed. The initial crawl that notified me it was done found the errors and were removed. Thanks for your help. Peter
Technical SEO | | VT_Pete0 -
Site Map Problems or Are They?
According to webmaster tools my Sitemap contains urls which are blocked by robots.txt Our site map is generically generated and encompasses all web pages, whether I have excluded them using the robots.txt file As far as I am aware this has never been an issue until recently. Is this hurting my rankings and how do I fix it? Secondly, webmaster tools says there is over 5,000 error/warnings on my site map. But site map is only 1,400 or so pages submitted. How do I see what is going on?
Technical SEO | | Professor0 -
Basic Multi-Site Question
Newb question. We run a site in multiple cities under the same domain. Often times one city will provide content that is "syndicated" to other cites. For example, here is the master post: http://www.styleblueprint.com/food-and-entertaining/kale-salad-quick-healthy/ The content will also show up in the following domains: http://atlanta.styleblueprint.com/food-and-entertaining/kale-salad-quick-healthy/ http://birmingham.styleblueprint.com/food-and-entertaining/recipes/kale-salad-quick-healthy/ Should I be marketing the posts in Atlanta and Birmingham as "no index, no follow" for SEO purposes? Thanks in advance, Jay
Technical SEO | | SSBCI0 -
Question about an older more obsolete site
I have a website that I don't use much anymore but it ranks on the first page for one of my main keywords. I am using another few websites in different niches right now that are doing better and are more functional. It may cost around 1,300 or so to get the website that I don't use anymore, to look and function in the new ways of the internet. Would you suggest that I: Do a site redesign (which is more difficult because to make the site do what I want it needs to be out of a wordpress theme) or 301 redirect the site to another one of my sites? Would it make sense to do a 301? The domain is 5 years old but doesn't bring in any leads anymore because it would take a redesign for that to happen. How can I still benefit from the SEO that I have done on that site? Thanks and sorry if this message is hard to follow. If I need to clear anything up please let me know.
Technical SEO | | blake-766240 -
Site forwarding - seo friendly or not?
Recently i decided to change my domain name - and although i have written several useful and working .htacess files with 301 redirects, this one became more complicated by the fact that I went through TWO domain name changes, before settling on the second one. Having seen some issues with the browser not being able to interpret correctly the .htaccess file, i temporarily suspended the .htaccess file, and opted instead for site forwarding. I don't know the mechanics behind site forwarding, or whether it is seo friendly or just a method for ip addressing, a sort of pseudo domain name server record change.
Technical SEO | | highersourcesites
I let it lie for a few weeks, until the dust settled, and yesterday put back the basic .htaccess file, with a 301 redirect, which directs the original domain name to be forwarded to the new one ( also it has a conditional in place to solve canonical issues). It works fine. But right now i am not seeing the link juice, the domain age, the domain page rank that it has. It has gone to zero, when it used to be three, sometimes four. I also made the change of address using webmaster tools. How long ( forever?) will it take to see my old page rank come back, even if it loses 10% from the change? And does site forwarding help or hinder seo ranking?0 -
Why did our site drop in Google rankings?
My site's URL (web address) is: http://tinyurl.com/3svn2l9 Hi there, We operate a travel site that lists numerous tours, accommodation and activities. Since 6th August 2011 we have dropped from top 10 SERP rankings of our pages to around result number 100 (page 10) and losing massive amount of visitors via Google Search. Our Yahoo and Bing rankings are still in the top10. We need your advice and quick! The last changes we have made are the following: -redirected the non-www version to the www version on the 1st August -bought advertising with a follow link in a sidebar that is being populated across the site (+4000 pages) about 2 months ago -added a blog to the website 2 weeks ago and posted 2 posts to date. Additionally, our website structure allows visitors (and bots) to see the same listings via different URLs which caused duplicate content. This has been the case since the launch of our website about 1 year ago. To prevent this duplicate content we have placed canonical tags on the individual listings pages. Why did our site all of a sudden plummet in the rankings?
Technical SEO | | Robbern0