Crawl rate drop
-
Hi Guys,
I have a crawl rate drop in webmastertools and can't figure out way. In the last month I removed a lot o duplicate pages that we don't need anymore, there were at least 1.5 million pages. Can this be a motive?
-
Hi there.
Of course it can! The crawl rate is in pages per day, so if you remove pages (especially 1.5 million), there won't be as much to crawl
Also it can happen, if you have the same static pages and the crawl bot has crawled them all. Google doesn't crawl all pages all the time, they have limited resources. So, if you have launched or updated website recently and now not really updating it, you can see the change in crawl rate.
However, you can change crawl rate for 90 days, if you need them to crawl your website constantly (usually good for websites in process of reconstruction). Here: https://support.google.com/webmasters/answer/48620?hl=en
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
ScreamingFrog won't crawl my site.
Hey guys, My site is Netspiren.dk and when I use a tool like Screaming Frog or Integrity, it only crawls my homepage and menu's - not product-pages. Examples
Intermediate & Advanced SEO | | FrederikTrovatten22
A menu: http://www.netspiren.dk/pl/Helse-Kosttilskud-Blandingsolie_57699.aspx
A product: http://www.netspiren.dk/pi/All-Omega-3-6-9-180-kapsler_1412956_57699.aspx Is it because the products are being loaded in Javascript?
What's your recommendation? All best,
Fred.0 -
Drop of 11 places - server issues?
Hi all, I have experienced a drop of 11 places for our main keyword since our last Moz rankings report which means page 2 to page 3. I have also received an error 904 in my weekly crawl report two weeks in a row and think this may be the reason for the drop. Could this result in a drop of 11 places? Last week I also tweeked the H1 tags on my main landing pages, nothing major just a reordering of the words. Could this result in such a major drop? Final question is does anyone know of any major algorithm update last week that could have affected rankings so drastically? Thanks all
Intermediate & Advanced SEO | | gavinr
Gavin0 -
Crawling issue
Hello, I am working on 3 weeks old new Magento website. On GWT, under index status >advanced, I can only see 1 crawl on the 4th day of launching and I don't see any numbers for indexed or blocked status. | Total indexed | Ever crawled | Blocked by robots | Removed |
Intermediate & Advanced SEO | | sedamiran
| 0 | 1 | 0 | 0 | I can see the traffic on Google Analytic and i can see the website on SERPS when i search for some of the keywords, i can see the links appear on Google but i don't see any numbers on GWT.. As far as I check there is no 'no index' or robot block issue but Google doesn't crawl the website for some reason. Any ideas why i cannot see any numbers for indexed or crawled status on GWT? Thanks Seda | | | | |
| | | | |0 -
How can Google index a page that it can't crawl completely?
I recently posted a question regarding a product page that appeared to have no content. [http://www.seomoz.org/q/why-is-ose-showing-now-data-for-this-url] What puzzles me is that this page got indexed anyway. Was it indexed based on Google knowing that there was once content on the page? Was it indexed based on the trust level of our root domain? What are your thoughts? I'm asking not only because I don't know the answer, but because I know the argument is going to be made that if Google indexed the page then it must have been crawlable...therefore we didn't really have a crawlability problem. Why Google index a page it can't crawl?
Intermediate & Advanced SEO | | danatanseo0 -
Why the sudden link drop?
A the end of November I am showing that our total links were 118k. Current links are 22k. We changed sites early November so that was about three weeks before. What would cause the drop of about 100k links? Or where should I start investigating?
Intermediate & Advanced SEO | | EcommerceSite0 -
Drop in Traffic on Friday April 20th.
Just curious if anyone noticed a drop in traffic last friday. I got hammered with about a 20% drop overall. Didn't know if there was an update or what. Thanks in advance!
Intermediate & Advanced SEO | | astahl110 -
202 error page set in robots.txt versus using crawl-able 404 error
We currently have our error page set up as a 202 page that is unreachable by the search engines as it is currently in our robots.txt file. Should the current error page be a 404 error page and reachable by the search engines? Is there more value or is it a better practice to use 404 over a 202? We noticed in our Google Webmaster account we have a number of broken links pointing the site, but the 404 error page was not accessible. If you have any insight that would be great, if you have any questions please let me know. Thanks, VPSEO
Intermediate & Advanced SEO | | VPSEO0 -
My site dropped in serp and backlinks ranked higher than it.
appliance-repair-ny.com I recently got some new good page rank backlinks (4-5) to my site. After a nice increase in rankings i left it for a couple of day (holiday) when i came back i found out that my rankings dropped dramatically for many different keywords when in some cases the backlinks are listed higher than me for the keyword. Any suggestions? should i keep on building links in the same paste, or slow down a little? UPDATE: I now noticed that the seomoz app has a warning that my homepage is missing title tag and meta tag. I'm using wordpress and headspace plugin. I recently updated wordpress.
Intermediate & Advanced SEO | | atohad0