How long will Google take to stop crawling an old URL once it has been 301 redirected
-
I need to do a clean-up old urls that have been redirected in sitemap and was wondering about this.
-
I agree with Ryan. As long as the old URLs are out there in the web, Google will follow them.
At the same time, redirected URL list will be very lengthy if you are maintaining a big site and maintaining them will consume your resources.
Check link value of the old URLs as a first step. If they have no value then you will be fine to retire such old URLs.
-
Google's crawler will always attempt to crawl any links it finds. If somewhere on the internet there is a link to an old URL and Google sees that link, it will never stop trying to explore that link. Each time it attempts to crawl the link, it will see the 301 and follow the redirect, but if you ever remove that redirect then the original URL will be seen.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 redirected all my pages to my new domain, now I have a problem with Google Search Console
Hi guys! I bought a new domain name and redirected all my URLs from the old domain to the new one. Everything worked perfectly but now I have a little problem. I want to use the option 'Address Change' in google search console. Step 1 Works (Select new website in the list) Step 2 Works (Confirm that the 301 are working) Step 3 Asks me to Verify the old domain (huh!?) in order to complete the request. Obviously that doesn't work because my 301s WORKS! So if I try to verify the old website by putting a google file in the root of my domain Google tries to access it and it automatically redirects to the new domain. I must be missing something lol help!
Technical SEO | | benoit_20180 -
Will a Robots.txt 'disallow' of a directory, keep Google from seeing 301 redirects for pages/files within the directory?
Hi- I have a client that had thousands of dynamic php pages indexed by Google that shouldn't have been. He has since blocked these php pages via robots.txt disallow. Unfortunately, many of those php pages were linked to by high quality sites mulitiple times (instead of the static urls) before he put up the php 'disallow'. If we create 301 redirects for some of these php URLs that area still showing high value backlinks and send them to the correct static URLs, will Google even see these 301 redirects and pass link value to the proper static URLs? Or will the robots.txt keep Google away and we lose all these high quality backlinks? I guess the same question applies if we use the canonical tag instead of the 301. Will the robots.txt keep Google from seeing the canonical tags on the php pages? Thanks very much, V
Technical SEO | | Voodak0 -
Why is my site crawled so much more in Bing then in Google?
I recently setup Cloudflare so I can see how much my site is being crawled. It looks like Bing is crawling me about 3 times as much as Google. Any ideas on why that would be or what I should check?
Technical SEO | | EcommerceSite0 -
When will all of Google Maps be the same again?
As many of you are aware that the pigeon update was only applied to the new Google maps resulting in very different search results for Google local business. When you search for a business on old Google maps then you get totally different results vs the new Google maps. Some businesses totally disappeared completely from the search results. I have done my research and found out that it's because the new Algo was only applied to the new maps. Also new algo does not apply to other countries. Well the reason I posted this topic is because I have noticed that all the new Google Business listings I am verifying for my clients are all being put under the old Google maps and not the new ones. They come up fine when searching from old maps but not the new ones. I understand Google has not rolled out the pigeon on all data centers but why? Will Google eventually roll out the update to old maps? Since Google is adding businesses to old google maps then what's the point of even adding new listings?
Technical SEO | | bajaseo0 -
Google disavow tool ( how long does it take ? )
Hello, I disavowed some of my links about three months but still see them in my link profile, using OSE? How long does it take for Google to make them nofollow. Thanks
Technical SEO | | mezozcorp0 -
How long should you keep 301 redirects?
Hi, Back in 2009 I decided to update an older site from .htm and .shtml to .php. In order to minimize the impact I would go in every month and do a 301 redirect on the .shtml page to the new .php page. So I have many that range from 2009 through 2010. I had left the old 301's because I felt they would only be used if needed but I would think I should clean up my .htaccess by removing the old 301 redirects if they are not needed. How long should you keep this type of 301 redirect? Thanks!
Technical SEO | | Force70 -
How can I redirect old URLs to new ones?
We are trying to optimize our website for search engines and one of the things we have noticed that needs to be changed are the URLs. I know how to modify the URLs but have no idea how I can get the old redirected to the new ones using 301. We have some pages ranking pretty high on Google and we can't loose this traffic. Our website is hosted on Plesk and it was coded using Cold Fusion. I do have access to the .htaccess file. Any help would be highly appreciated.
Technical SEO | | personalproducts0 -
301 redirects
Hi Guys, Question,
Technical SEO | | VividLime
Lets say I have a page oldfile.php at position #2 then set-up a redirection in the following way 100 incoming external links--> oldfile.php [301 to] newfile.php Google comes along and updates its index to newfile.php and ranking of newfile.php remains at position #2. Everything is good. Lets say in 5months, I come along and delete oldfile.php so we have
100 incoming external links--> deleted(oldfile.php) or 100 incoming external links-->404 error. |||| newfile.php Do I then loose the rankings on newfile.php. My thinking is that now that all the external links now point to a page not found, newfile.php should loose rankings Am I correct in my assumption?0