Does Google make continued attempts to crawl an old page one it has followed a 301 to the new page?
-
I am curious about this for a couple of reasons.
We have all dealt with a site who switched platforms and didn't plan properly and now have 1,000's of crawl errors. Many of the developers I have talked to have stated very clearly that the HTacccess file should not be used for 1,000's of singe redirects. I figured If I only needed them in their temporarily it wouldn't be an issue. I am curious if once Google follows a 301 from an old page to a new page, will they stop crawling the old page?
-
It will keep on crawling. I have seen 6 months to 1 year later they come back. Especially if there are links from other websites pointing to your old URLs.
Why they do repeat crawls on 301s and 404s for that matter is that they know how much the web changes and assume that there may have been a mistake and so want to double check. Also, you have to consider, they will not recrawl 1000s of pages all at once. They will probably pick up the most popular URLs first as they crawl them more often and then the less popular pages later.
IMHO. You will need to find a long term solution in place.
Here is the why you dont want to use htaccess
http://stackoverflow.com/questions/209209/htaccess-redirect-performance
One suggestion for Apache - RewriteMap
http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html#rewritemap
That or all the developers who are telling you not to use .htaccess should have at least 2 solutions to use instead that should work. Go back to them to solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google indexing wrong pages
We have a variety of issues at the moment, and need some advice. First off, we have a HUGE indexing issue across our entire website. Website in question: http://www.localsearch.com.au/ Firstly
Intermediate & Advanced SEO | | localdirectories
In Google.com.au, if you search for 'plumbers gosford' (https://www.google.com.au/#q=plumbers+gosford), the wrong page appears - in this instance, the page ranking should be http://www.localsearch.com.au/Gosford,NSW/Plumbers I can see this across the board, across multiple locations. Secondly
Recently I've seen Google reporting in 'Crawl Errors' in webmaster tools URLs such as:
http://www.localsearch.com.au/Saunders-Beach,QLD/Electronic-Equipment-Sales-Repairs&Sa=U&Ei=xs-XVJzAA9T_YQSMgIHQCw&Ved=0CIMBEBYwEg&Usg=AFQjCNHXPrZZg0JU3O4yTGjWbijon1Q8OA This is an invalid URL, and more specifically, those query strings seem to be referrer queries from Google themselves: &Sa=U&Ei=xs-XVJzAA9T_YQSMgIHQCw&Ved=0CIMBEBYwEg&Usg=AFQjCNHXPrZZg0JU3O4yTGjWbijon1Q8OA Here's the above example indexed in Google: https://www.google.com.au/#q="AFQjCNHXPrZZg0JU3O4yTGjWbijon1Q8OA" Does anyone have any advice on those 2 errors?0 -
Do I eventually 301 a page on our site that "expires," to a page that's related, but never expires, just to utilize the inbound link juice?
Our company gets inbound links from news websites that write stories about upcoming sporting events. The links we get are pointing to our event / ticket inventory pages on our commerce site. Once the event has passed, that event page is basically a dead page that shows no ticket inventory, and has no content. Also, each “event” page on our site has a unique url, since it’s an event that will eventually expire, as the game gets played, or the event has passed. Example of a url that a news site would link to: mysite.com/tickets/soldier-field/t7493325/nfc-divisional-home-game-chicago bears-vs-tbd-tickets.aspx Would there be any negative ramifications if I set up a 301 from the dead event page to another page on our site, one that is still somewhat related to the product in question, a landing page with content related to the team that just played, or venue they play in all season. Example, I would 301 to: mysite.com/venue/soldier-field tickets.aspx (This would be a live page that never expires.) I don’t know if that’s manipulating things a bit too much.
Intermediate & Advanced SEO | | Ticket_King1 -
Old pages still in index
Hi Guys, I've been working on a E-commerce site for a while now. Let me sum it up : February new site is launched Due to lack of resources we started 301's of old url's in March Added rel=canonical end of May because of huge index numbers (developers forgot!!) Added noindex and robots.txt on at least 1000 urls. Index numbers went down from 105.000 tot 55.000 for now, see screenshot (actual number in sitemap is 13.000) Now when i do site:domain.com there are still old url's in the index while there is a 301 on the url since March! I know this can take a while but I wonder how I can speed this up or am doing something wrong. Hope anyone can help because I simply don't know how the old url's can still be in the index. 4cArHPH.png
Intermediate & Advanced SEO | | ssiebn70 -
What is the value of Google Crawling Dynamic URLS with NO SEO
Hi All I am Working on travel site for client where there are 1000's of product listing pages that are dynamically created. These pages are not SEO optimised and are just lists of products with no content other than the product details. There are no meta tags for title and description on the listings pages. You then click Find Out more to go to the full product details. There is no way to SEO these Dynamic pages This main product details has no content other than details and now meta tags. To help increase my google rankings for the rest of the site which is search optimised would it be better to block google from indexing these pages. Are these pages hurting my ability to improve rankings if my SEO of the content pages has been done to a good level with good unique Titles, descriptions and useful content thanks In advance John
Intermediate & Advanced SEO | | ingageseo0 -
Will Google bots crawl tablet optimized pages of our site?
We are in the process of creating a tablet experience for a portion of our site. We haven’t yet decided if we will use a one URL structure for pages that will have a tablet experience or if we will create separate URLs that can only be access by tablet users. Either way, will the tablet versions of these pages/URLs be crawled by Google bots?
Intermediate & Advanced SEO | | kbbseo0 -
Google Penalty, Which one is hitting me?
Hello there SEOMOZ'ers, Just signed up for a Pro Account. Impressed with the trial. Just wondering is there any way to find out if you have fallen foul of a Google update, is there any way to know which you have fallen foul of or is the date it happens the only guide or pointer to what might have happened? My site was getting a reasonable amount of traffic for the last few years and was ranking well for my main search terms up till Feburary the 4th when the traffic fell by almost 50% ... and it has never really recovered. I thought it might be the Above the Fold Advertising penalty (or whatever it is called) because I had a lot of images linking elsewhere on my site, above the fold on my homepage, but I removed them about 4 weeks ago and the traffic still hasnt recovered. I've been using SEOMOZ to detect duplicate content and various other problems with my site since and fixing a good few of them each week. I have the main issues down from about 300 to about 20 which i think i have just fixed. I have also used Copyscape and found that some other websites had stolen some of my content, which i have since rewritten. I was ranking number 2 or 3 for my main keyword but dropped out of the top 100 in February, which is pretty frustrating at this stage. I don;t know if I have accidentally gone too far over optimising my site or what has happened. If anyone can give me some pointers I'd really appreciate it, if anyone can help me fix it I'll buy them a beer next time you are in Ireland! 🙂 Thanks, Shane
Intermediate & Advanced SEO | | shaneod0 -
How to Optimize Product Page for Better Ranking in Google?
Today, I was searching for product page SEO on Google for better optimization of product pages and category level pages. I want to introduce about my website structure. Root category: http://www.vistastores.com/outdoor Sub category: http://www.vistastores.com/outdoor-umbrellas End level category: http://www.vistastores.com/patio-umbrellas Product page: http://www.vistastores.com/patio-umbrellas-california-umbrella-slpt758-f13-red.html I'm doing link building for sub category & end level category page. But, I'm not able to do inbound marketing for product level URLs due to resource problem. But, I have checked that... There are many competitors who are rank well with long trail keyword... I'm selling same product but not ranking well... What is reason behind it? I don't know... Just look at Olefin Red Patio Umbrella search result. I want to rank in top 3 with this kind of long trail keywords... I have made study of following video and article. http://www.seomoz.org/blog/ecommerce-seo-making-product-pages-into-great-content-whiteboard-friday http://www.distilled.net/blog/seo/in-search-of-the-perfect-seo-product-page/ I have done all things which were described by them... But, I'm still waiting for good ranking...
Intermediate & Advanced SEO | | CommercePundit0
