Bots gone from site for 1-2 hours
-
Hi everyone,
From the beginning of March bots and crawlers begin to disappear from site completely for 1-2 hours and then come back again. It never happened earlier, they constantly crawled our site without stopping. We didn't make any changes on the site.
After that pages and images start to disappear from G index.
Does anyone have the same trouble?
-
Assuming you are talking about the website in your profile...here is what I would check: Do you have any content duplication between your multiple language subdomains ? Do you have XML Sitemaps for your pages ? Are they submitted via the Webmaster Console ? If yes, what %age of your pages do you see indexed vs how many you submitted ? Have you considered doing an Images XML Sitemap ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Our clients Magento 2 site has lots of obsolete categories. Advice on SEO best practice for setting server level redirects so I can delete them?
Our client's Magento website has been running for at least a decade, so has a lot of old legacy categories for Brands they no longer carry. We're looking to trim down the amount of unnecessary URL Redirects in Magento, so my question is: Is there a way that is SEO efficient to setup permanent redirects at a server level (nginx) that Google will crawl to allow us at some point to delete the categories and Magento URL Redirects? If this is a good practice can you at some point then delete the server redirects as google has marked them as permanent?
Technical SEO | | WillyGx0 -
Representing categories on my site
My site serves a consumer-focused industry that has about 15-20 well recognized categories, which act as a pretty obvious way to segment our content. Each category supports it's own page (with some useful content) and a series of articles relevant to that category. In short, the categories are pretty focal to what we do. I am moving from DNN to WordPress as my CMS/blog. I am taking the opportunity to review and fix SEO-related issues as I migrate. One such area is my URL structure. On my existing site (on DNN), I have the following types of pages for each topic: / <topic>- this is essentially the landing page for the topic and links to articles</topic> /<topic>/articles/ <article-name>- topics have 3-15 articles with this URL structure</article-name></topic> With WordPress, I am considering moving to articles being under the root. So, an article on (making this up) how to make a widget would be under /how-to-make-a-widget, instead of /<widgets>/article/how-to-make-a-widget I will be using WordPress categories to reflect the topics taxonomy, so I can flag my articles using standard WordPress concepts.</widgets> Anyway, I'm trying to get my head around whether it makes sense to "flatten" my URL structure such that the URLs for each article no longer include the topic (the article page will link to the topic page though). Thoughts?
Technical SEO | | MarkWill1 -
No follow links from ban sites
Hello boys and girls. I'm auditing my link profile and came across links pointing to my sites, from a ban site (site: domain.com gives back no result).yet the links are no follow. should i try to remove them all? i know no follow should be enough, yet my links are set in a bad neighborhood. what would you recommend??
Technical SEO | | Tit0 -
Should we dump the https from a client site?
We inherited a site that has both http and https. No e-commerce or data transfer...just html. Should we dump the https certificate? I think it might be causing issues with indexing and possible duplicate content. The https site has a certificate warning message...not good. The URL is www.charlottemechanical.com
Technical SEO | | theideapeople0 -
Site being indexed by Google before it has launched
We are currently coming towards the end of a site migration, and are at the final stage of testing redirects etc. However, to our horror we've just discovered Google has started indexing the new site. Any ideas on how this could have happened? I have most recently asked for robots.txt to exclude anything with a certain parameter in URL. Is there a chance this, wrongly implemented, could have caused this?
Technical SEO | | Sayers0 -
How to find all the links to my site
hi i have been trying to find all the links that i have to my site http://www.clairehegarty.co.uk but i am not having any luck. I have used the open explorer but it is not showing all the links but when i go to my google webmaster page it shows me more pages than it does on the semoz tool. can anyone help me sort this out and find out exactly what links are going into my site many thanks
Technical SEO | | ClaireH-1848860 -
See your sites Architecture
Does anybody know a problem where you can see how your internal linkings look to the search engines?
Technical SEO | | ScottBaxterWW0 -
My site has vanished from google
Hi my site has vanished from google. We have been for a very long time. for example if you put in gastric band hypnotherapy then we would be first page number two and also lots of other keywords but now we have vanished from google and i do not know why or how to solve this. can anyone please help me and help me understand what i need to do to solve this please My site is http://www.clairehegarty.co.uk I am not sure if i have been banned or why i have dropped out of google
Technical SEO | | ClaireH-1848860