Crawl rate drop
-
Hi Guys,
I have a crawl rate drop in webmastertools and can't figure out way. In the last month I removed a lot o duplicate pages that we don't need anymore, there were at least 1.5 million pages. Can this be a motive?
-
Hi there.
Of course it can! The crawl rate is in pages per day, so if you remove pages (especially 1.5 million), there won't be as much to crawl
Also it can happen, if you have the same static pages and the crawl bot has crawled them all. Google doesn't crawl all pages all the time, they have limited resources. So, if you have launched or updated website recently and now not really updating it, you can see the change in crawl rate.
However, you can change crawl rate for 90 days, if you need them to crawl your website constantly (usually good for websites in process of reconstruction). Here:Â https://support.google.com/webmasters/answer/48620?hl=en
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My last site crawl shows over 700 404 errors all with void(0 added to the ends of my posts/pages.
Hello, My last site crawl shows over 700 404 errors all with void(0 added to the ends of my posts/pages. I have contacted my theme company but not sure what could have done this. Any ideas? The original posts/pages are still correct and working it just looks like it did duplicates and added void(0 to the end of each post/page. Questions: There is no way to undo this correct? Do I have to do a redirect on each of these? Will this hurt my rankings and domain authority? Any suggestions would be appreciated. Thanks, Wade
Intermediate & Advanced SEO | | neverenoughmusic.com0 -
Does google still not crawl forms with a method=post?
I know back in 08 Google started crawling forms using the method=get however not method=post. whats the latest? is this still valid?
Intermediate & Advanced SEO | | Turkey0 -
How does Tripadviser ensure all their user reviews get crawled?
Tripadvisor has a LOT of user generated content. Searching for a random hotel always seems to return a paginated list of 90+ pages. However once the first page is clicked and "#REVIEWS" is appended to the URL, the URL never changes with any subsequent clicks of the paginated links. How do they ensure that all this review content gets crawled? Thanks, linklater
Intermediate & Advanced SEO | | linklater0 -
Can changing G+ authorship on a well-ranking article drop its search ranking?
We have an article that ranks #1 in Google SERP for the keyword we want it to rank for. We decided to revise the article because although it's performing well, we knew it could be better and more informative for the user. Now that we've revised the content, we're wondering: Should we update the article author (and the G+ authorship markup) to reflect that the revisor authored the content, or keep the original author listed? Can changing G+ authorship on an article impact its search ranking, or is that an issue that's a few Google algorithm updates down the road?
Intermediate & Advanced SEO | | pasware0 -
Does anyone know why my website's domain authority has dropped from 51 to 49
However this does not seem to be in isolation. All of my competitors websites have taken a similar 1 or 2 points hit. I am thinking that as an industry we may have been affected by a mutual linking site being taken down, redesigned or just loosing its own domain authority. We do rank well for our keywords and we have been on a continual rise since I took over in January, we do a little bit of a guest blogging and I am trying to build links to the site but I am doing it slowly. Would anyone else have an idea on what has happened that would cause 4 sites in the same industry to take a 1 or 2 point hit? Thanks, Emmet
Intermediate & Advanced SEO | | CertificationEU1 -
Meta No INDEX and Robots - Optimizing Crawl Budget
Hi, Sometime ago, a few thousand pages got into Google's index - they were "product pop up" pages, exact duplicates of the actual product page but a "quick view". So I deleted them via GWT and also put in a Meta No Index on these pop up overlays to stop them being indexed and causing dupe content issues. They are no longer within the index as far as I can see, i do a site:www.mydomain.com/ajax and nothing appears - So can I block these off now with robots.txt to optimize my crawl budget? Thanks
Intermediate & Advanced SEO | | bjs20100 -
UK Company Major drop in traffic & rankings on one primary keyword since March
I am helping out a small UK company who have had a sudden drop in organic search traffic since March 24th. Investigation highlights some issues with the site,e.g. Potential canonicalization of home page, a few html errors, some inbound links to the /index.html version of the homepage rather than /. But, nothing particualrly major and nothing that is different to pre-March 24th. The indexed pages looks ok in Google (although Bing is ranking the non-www version of the homepage) but this does not appear in Google's index. Searches for the company name on Google.co.uk show it as top result & some keywords are ranking reasonably well (based on homepage). Selecting blocks of text from the homepage and it ranks #1, but its Google rank for the primary keyword has gone from #2 pre-March 24th to not in the top 100 results since. SEOMOZ is grading the page A for the keyword which appears prominently on the page & keyword is the first characters of the title. It is not a particularly competitive keyword. Adding UK to the keyword and the page is Google.co.uk ranked #3. It's almost as if they are being penalised for a single keyword which I've never seen or heard of before. Any ideas? ** The company has never carried out any SEO - white hat or black hat. The site is perfectly normal, nothing dodgy or concerning about it at all.** Thanks in advance for your advice.
Intermediate & Advanced SEO | | bjalc20110 -
Negative impact on crawling after upload robots.txt file on HTTPS pages
I experienced negative impact on crawling after upload robots.txt file on HTTPS pages. You can find out both URLs as follow. Robots.txt File for HTTP: http://www.vistastores.com/robots.txt Robots.txt File for HTTPS: https://www.vistastores.com/robots.txt I have disallowed all crawlers for HTTPS pages with following syntax. User-agent: *
Intermediate & Advanced SEO | | CommercePundit
Disallow: / Does it matter for that? If I have done any thing wrong so give me more idea to fix this issue.0
