Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Would you rate-control Googlebot? How much crawling is too much crawling?
-
One of our sites is very large - over 500M pages. Google has indexed 1/8th of the site - and they tend to crawl between 800k and 1M pages per day.
A few times a year, Google will significantly increase their crawl rate - overnight hitting 2M pages per day or more. This creates big problems for us, because at 1M pages per day Google is consuming 70% of our API capacity, and the API overall is at 90% capacity. At 2M pages per day, 20% of our page requests are 500 errors.
I've lobbied for an investment / overhaul of the API configuration to allow for more Google bandwidth without compromising user experience. My tech team counters that it's a wasted investment - as Google will crawl to our capacity whatever that capacity is.
Questions to Enterprise SEOs:
*Is there any validity to the tech team's claim? I thought Google's crawl rate was based on a combination of PageRank and the frequency of page updates. This indicates there is some upper limit - which we perhaps haven't reached - but which would stabilize once reached.
*We've asked Google to rate-limit our crawl rate in the past. Is that harmful? I've always looked at a robust crawl rate as a good problem to have.
- Is 1.5M Googlebot API calls a day desirable, or something any reasonable Enterprise SEO would seek to throttle back?
*What about setting a longer refresh rate in the sitemaps? Would that reduce the daily crawl demand? We could set increase it to a month, but at 500M pages Google could still have a ball at the 2M pages/day rate.
Thanks
-
I agree with Matt that there can probably be a reduction of pages, but that aside, how much of an issue this is comes down to what pages aren't being indexed. It's hard to advise without the site, are you able to share the domain? If the site has been around for a long time, that seems a low level of indexation. Is this a site where the age of the content matters? For example Craigslist?
Craig
-
Thanks for your response. I get where you're going with that. (Ecomm store gone bad.) It's not actually an Ecomm FWIW. And I do restrict parameters - the list is about a page and a half long. It's a legitimately large site.
You're correct - I don't want Google to crawl the full 500M. But I do want them to crawl 100M. At the current crawl rate we limit them to, it's going to take Google more than 3 months to get to each page a single time. I'd actually like to let them crawl 3M pages a day. Is that an insane amount of Googlebot bandwidth? Does anyone else have a similar situation?
-
Gosh, that's a HUGE site. Are you having Google crawl parameter pages with that? If so, that's a bigger issue.
I can't imagine the crawl issues with 500M pages. A site:amazon.com search only returns 200M. Ebay.com returns 800M so your site is somewhere in between these two? (I understand both probably have a lot more - but not returning as indexed.)
You always WANT a full site crawl - but your techs do have a point. Unless there's an absolutely necessary reason to have 500M indexed pages, I'd also seek to cut that to what you want indexed. That sounds like a nightmare ecommerce store gone bad.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
XML sitemap generator only crawling 20% of my site
Hi guys, I am trying to submit the most recent XML sitemap but the sitemap generator tools are only crawling about 20% of my site. The site carries around 150 pages and only 37 show up on tools like xml-sitemaps.com. My goal is to get all the important URLs we care about into the XML sitemap. How should I go about this? Thanks
Intermediate & Advanced SEO | | TyEl0 -
My last site crawl shows over 700 404 errors all with void(0 added to the ends of my posts/pages.
Hello, My last site crawl shows over 700 404 errors all with void(0 added to the ends of my posts/pages. I have contacted my theme company but not sure what could have done this. Any ideas? The original posts/pages are still correct and working it just looks like it did duplicates and added void(0 to the end of each post/page. Questions: There is no way to undo this correct? Do I have to do a redirect on each of these? Will this hurt my rankings and domain authority? Any suggestions would be appreciated. Thanks, Wade
Intermediate & Advanced SEO | | neverenoughmusic.com0 -
How do I know if I am correctly solving an uppercase url issue that may be affecting Googlebot?
We have a large e-commerce site (10k+ SKUs). https://www.flagandbanner.com. As I have begun analyzing how to improve it I have discovered that we have thousands of urls that have uppercase characters. For instance: https://www.flagandbanner.com/Products/patriotic-paper-lanterns-string-lights.asp. This is inconsistently applied throughout the site. I directed our website vendor to fix the issue and they placed 301 redirects via a rule to the web.config file. Any url that contains an uppercase character now displays as a lowercase. However, as I use screaming frog to monitor our site, I see all these 301 redirects--thousands of them. The XML sitemap still shows the the uppercase versions. We have had indexing issues as well. So I'm wondering what is the most effective way to make sure that I'm not placing an extra burden on Googlebot when they index our site? Should I have just not cared about the uppercase issue and let it alone?
Intermediate & Advanced SEO | | webrocket0 -
Mobile Googlebot vs Desktop Googlebot - GWT reports - Crawl errors
Hi Everyone, I have a very specific SEO question. I am doing a site audit and one of the crawl reports is showing tons of 404's for the "smartphone" bot and with very recent crawl dates. If our website is responsive, and we do not have a mobile version of the website I do not understand why the desktop report version has tons of 404's and yet the smartphone does not. I think I am not understanding something conceptually. I think it has something to do with this little message in the Mobile crawl report. "Errors that occurred only when your site was crawled by Googlebot (errors didn't appear for desktop)." If I understand correctly, the "smartphone" report will only show URL's that are not on the desktop report. Is this correct?
Intermediate & Advanced SEO | | Carla_Dawson0 -
After Server Migration - Crawling Gets slow and Dynamic Pages wherein Content changes are not getting Updated
Hello, I have just performed doing server migration 2 days back All's well with traffic moved to new servers But somehow - it seems that w.r.t previous host that on submitting a new article - it was getting indexed in minutes. Now even after submitting page for indexing - its taking bit of time in coming to Search Engines and some pages wherein content is daily updated - despite submitting for indexing - changes are not getting reflected Site name is - http://www.mycarhelpline.com Have checked in robots, meta tags, url structure - all remains well intact. No unknown errors reports through Google webmaster Could someone advise - is it normal - due to name server and ip address change and expect to correct it automatically or am i missing something Kindly advise in . Thanks
Intermediate & Advanced SEO | | Modi0 -
ScreamingFrog won't crawl my site.
Hey guys, My site is Netspiren.dk and when I use a tool like Screaming Frog or Integrity, it only crawls my homepage and menu's - not product-pages. Examples
Intermediate & Advanced SEO | | FrederikTrovatten22
A menu: http://www.netspiren.dk/pl/Helse-Kosttilskud-Blandingsolie_57699.aspx
A product: http://www.netspiren.dk/pi/All-Omega-3-6-9-180-kapsler_1412956_57699.aspx Is it because the products are being loaded in Javascript?
What's your recommendation? All best,
Fred.0 -
Cache-Control max-age=3, must-revalidate
Good morning, I am using wp super cache and I got the report: Cache-Control max-age=3, must-revalidate Any idea how to fix this? Thank you very much for any advice. My htacces file look like this below: EXPIRES CACHING <ifmodule mod_expires.c="">ExpiresActive On
Intermediate & Advanced SEO | | Rebeca1
ExpiresByType image/jpg "access plus 1 week"
ExpiresByType image/jpeg "access plus 1 week"
ExpiresByType image/gif "access plus 1 week"
ExpiresByType image/png "access plus 1 week"
ExpiresByType text/css "access plus 1 week"
ExpiresByType application/pdf "access plus 1 week"
ExpiresByType text/x-javascript "access plus 1 week"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 week"
ExpiresDefault "access plus 2 days"</ifmodule> EXPIRES CACHING BEGIN s2Member GZIP exclusions Redirect 301 /2012/03/22/romantic-couples-getaway-on-diani-beach/ http://www.villasdiani.com/rent/alfajiri-beach-villa/
Redirect 301 /luxury-beach-holidays-in-kenya/ http://www.villasdiani.com/
Redirect 301 /diani-beach-family-villa/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rentals/ocean-view-villas/ http://www.villasdiani.com/rent/alfajiri-beach-villa/
Redirect 301 /alfajiri-garden-villa/alfajiri-cliff-villa-diani-beach-4-2/feed/ http://www.villasdiani.com/rent/alfajiri-beach-villa/
Redirect 301 /kenyas-guide-highlights-activities/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /2012/06/25/safar-activities-tours-in-from-diani-beach/ http://www.villasdiani.com/category/diani-beach-restaurants-bars/
Redirect 301 /afrochic-special-offer/ http://www.villasdiani.com/rent/afrochic-boutique-hotel/
Redirect 301 /rent/african-beach-cottages/ http://www.villasdiani.com/rentals/boutique-hotels/
Redirect 301 /2012/07/11/boutique-hotels-in-kenya/ http://www.villasdiani.com/rentals/boutique-hotels/
Redirect 301 /tag/north-coast-accommodation/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /tag/coast-weather/ http://www.villasdiani.com/kenya-coast/
Redirect 301 /category/top-destination-guide/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /watamu-tembo-village-restaurant/ http://www.villasdiani.com/category/diani-beach-restaurants-bars/
Redirect 301 /kikambala-beach/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /kilifi-bofa-maweni-beach/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /rent/exclusive-beachfront-holiday-villa-diani-beach/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /news/ http://www.villasdiani.com/category/kenya-news/
Redirect 301 /rent/galu-beach-beachfront-cottages/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /wp-content/uploads/2012/05/Diani-beach-ocean-view-300x200.jpg http://www.villasdiani.com/diani-beach/
Redirect 301 /wp-content/uploads/2012/05/Diani-beach-ocean-view.jpg http://www.villasdiani.com/diani-beach/
Redirect 301 /feature/ocean-view/ http://www.villasdiani.com/rentals/ocean-view-villas/
Redirect 301 /rent/forodhani-house/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/al-hamra-residence/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/lonno-lodge-watamu/ http://www.villasdiani.com/rentals/boutique-hotels/
Redirect 301 /rent/blue-bay-cove-watamu/ http://www.villasdiani.com/rentals/boutique-hotels/
Redirect 301 /rent/msambweni-beach-house/ http://www.villasdiani.com/rentals/boutique-hotels/
Redirect 301 /rent/diamonds-dream-of-africa/ http://www.villasdiani.com/rentals/boutique-hotels/
Redirect 301 /rent/papaya-garden/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /category/activities/ http://www.villasdiani.com/category/diani-beach-safaris-tours/
Redirect 301 /category/restaurants-and-nightclubs-diani-beach/ http://www.villasdiani.com/category/diani-beach-restaurants-bars/
Redirect 301 /category/restaurants-nightclubs/ http://www.villasdiani.com/category/diani-beach-restaurants-bars/
Redirect 301 /diani-beach-hospital/ http://www.villasdiani.com/diani-beach-hospitals/
Redirect 301 /health-care-facility-diani/ http://www.villasdiani.com/diani-beach-hospitals/
Redirect 301 /palm-beach-hospital/ http://www.villasdiani.com/diani-beach-hospitals/
Redirect 301 /category/news/kenyas-beaches/page/2/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /2011/09/27/unique-private-palatial-diani-beach-villa/ http://www.villasdiani.com/rent/presidential-villa/
Redirect 301 /kenya-coast-weather-forecast/ http://www.villasdiani.com/kenya-coast/
Redirect 301 /rent/beachfront-tropical-paradise-accommodation/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rentals/villa-accommodation/page/3/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /category/restaurants-and-nightclubs/page/2/ http://www.villasdiani.com/category/diani-beach-restaurants-bars/
Redirect 301 /rentals/villa-accommodation/page/4/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rentals/villa-accommodation/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /free-night-afrochic/ http://www.villasdiani.com/rent/afrochic-boutique-hotel/
Redirect 301 /flamboyant-diani-special-offer/ http://www.villasdiani.com/rent/flamboyant-boutique-hotel/
Redirect 301 /restaurants-night-clubs-diani-beach/ http://www.villasdiani.com/category/diani-beach-restaurants-bars/
Redirect 301 /category/beaches-kenya/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /rent/luxury-beach-cottage-firimbi/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /safaris/ http://www.villasdiani.com/rentals/safari-villas/
Redirect 301 /sitemap/ http://www.villasdiani.com/sitemap_index.xml
Redirect 301 /holiday-accommodation-and-lodging-in-kenya/ http://www.villasdiani.com/holiday-accommodation-kenya/
Redirect 301 /beach-holidays-to-mombasa/ http://www.villasdiani.com/mombasa/
Redirect 301 /wp-content/uploads/2012/07/luxury-holiday-villa-300x200.jpg http://www.villasdiani.com/kenya-holidays/
Redirect 301 /wp-content/uploads/2012/07/beach-family1-300x200.jpg http://www.villasdiani.com/holiday-accommodation-kenya/
Redirect 301 /kenya-luxury-holidays/ http://www.villasdiani.com/kenya-holidays/
Redirect 301 /rent/flamboyant-diani-hotel/ http://www.villasdiani.com/rent/flamboyant-boutique-hotel/
Redirect 301 /rent/galu-beach-kenya-luxury-boutique-hotel/ http://www.villasdiani.com/rent/galu-beach-hotel/
Redirect 301 /rent/romantic-couples-getaway-on-diani-beach/ http://www.villasdiani.com/rent/spice-of-the-coast/
Redirect 301 /rent/diani-beach-beachfront-boutique-resort/ http://www.villasdiani.com/rent/waterlovers-beach-resort/
Redirect 301 /rent/galu-beach-tropical-paradise-villa/ http://www.villasdiani.com/rent/paradise-villas/
Redirect 301 /rent/galu-beach-beachfront-cottages/ http://www.villasdiani.com/property-type/self-catering/
Redirect 301 /rentals/diani-beach-holiday-villas/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rentals/private-villas/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/diani-beach-villa/ http://www.villasdiani.com/rent/sofia-house/
Redirect 301 /rent/diani-beach-family-villa/ http://www.villasdiani.com/rent/satis-house/
Redirect 301 /rent/diani-beach-zanzibar-style-villa/ http://www.villasdiani.com/rent/taj-riviera/
Redirect 301 /rent/diani-beach-majestic-arabian-villa/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/diani-beach-congo-river-villa/ http://www.villasdiani.com/rent/congo-river-house/
Redirect 301 /rent/galu-beach-cottage/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/diani-beach-central-villa/ http://www.villasdiani.com/rent/cinders-holiday-home/
Redirect 301 /rent/beachfront-villa-apartment-resort/ http://www.villasdiani.com/rent/lantana-galu-beach-resort/
Redirect 301 /rent/diani-beach-villa-hotel/ http://www.villasdiani.com/rent/afrochic-boutique-hotel/
Redirect 301 /rent/exotic-tree-villas-private-beach/ http://www.villasdiani.com/rent/cove-retreat/
Redirect 301 /rent/luxury-watamu-villa-yin-yang/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/lamu-luxury-beach-villa/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/watamu-exclusive-residence-al-hamra/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /rent/diani-beach-luxury-baobab-villas/ http://www.villasdiani.com/rent/adansonia-villas-resort/
Redirect 301 /rent/diani-beach-palatial-villa/ http://www.villasdiani.com/rent/presidential-villa/
Redirect 301 /rent/entiwi-beach-exclusive-beach-villa/ http://villasdiani.com/rent/baobab-house/
Redirect 301 /rent/diani-beach-luxury-beachfront-holiday-home/ http://www.villasdiani.com/rent/ocean-view-villa/
Redirect 301 /rent/exclusive-holiday-beach-villa-diani-beach/ http://www.villasdiani.com/rent/niros-paradise/
Redirect 301 /rent/luxury-beachfront-holiday-villa-diani-beach/ http://www.villasdiani.com/rent/niros-place/
Redirect 301 /rent/7-bedroom-diani-beach-beachfront-villa/ http://www.villasdiani.com/rent/watano-house/
Redirect 301 /rent/luxurious-diani-beach-villa-resort/ http://www.villasdiani.com/rent/almanara-beach-resort/
Redirect 301 /rent/tiwi-beachoceanfront-villa/ http://www.villasdiani.com/rent/waterside-villa/
Redirect 301 /rent/diani-beach-galu-beach-villa/ http://www.villasdiani.com/rent/sunset-villa/
Redirect 301 /wp-content/uploads/2012/06/vilan-300x225.jpg http://www.villasdiani.com/kitesurfing-windsurfing/
Redirect 301 /10-villa-cottage-diani-beach/ http://www.villasdiani.com/rentals/beach-villas/
Redirect 301 /small-luxury-hotels-kenya/ http://www.villasdiani.com/rentals/boutique-hotels/
Redirect 301 /beach-holidays-kenya/ http://www.villasdiani.com/kenya-beach-holidays/
Redirect 301 /baboons-in-diani-beach/ http://www.villasdiani.com/baby-baboon-video/
Redirect 301 /safaris-kenya/ http://www.villasdiani.com/national-parks/
Redirect 301 /diani-beach-big-game-fishing-deep-sea-fishing/ http://www.villasdiani.com/sport-fishing/
Redirect 301 /funzi-island-visit-diani-beach-sundowner-cruise/ http://www.villasdiani.com/sundowner-cruise/
Redirect 301 /diani-beach-beachfront-luxury-cottage/ http://www.villasdiani.com/rent/summer-villa-colobus/
Redirect 301 /highlights-activities-places-kenya/ http://www.villasdiani.com/safari-tours/
Redirect 301 /eco-bike-cultural-tour-on-diani-beach/ http://www.villasdiani.com/diani-bikes/
Redirect 301 /wasini-island-visit/ http://www.villasdiani.com/wasini-island/
Redirect 301 /diani-maasai-mara/ http://www.villasdiani.com/safari-beach-holidays-kenya/
Redirect 301 /geography-kenya-map-kenya/ http://www.villasdiani.com/where-is-kenya/
Redirect 301 /book-luxury-holiday-accommodation/ http://www.villasdiani.com/about-us-why-booking-your-holiday-with-us/
Redirect 301 /kitesurfing-windsurfing-in-diani-beach/ http://www.villasdiani.com/kitesurfing-windsurfing/
Redirect 301 /diani-beach-ali-barbours-cave-restaurant/ http://www.villasdiani.com/cave-restaurant/
Redirect 301 /diani-beach-forty-thieves-beach-bar/ http://www.villasdiani.com/forty-thieves-bar/
Redirect 301 /exotic-luxury-holiday-packages/ http://www.villasdiani.com/safari-beach-holidays-kenya/
Redirect 301 /category/news/ http://www.villasdiani.com/category/kenya-news/
Redirect 301 /category/kenya/ http://www.villasdiani.com/category/kenya-news/
Redirect 301 /vipingo-beach-bureni-beach/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /category/diani-beach/ http://www.villasdiani.com/diani-beach/
Redirect 301 /category/beaches/ http://www.villasdiani.com/beaches-in-kenya/
Redirect 301 /diani-beach-aniellos-restaurant/ http://www.villasdiani.com/aniello-restaurante/
Redirect 301 /diani-beach-shopping-areas-supermarkets-grocery-shops-local-markets/ http://www.villasdiani.com/diani-beach-shopping-local-markets/
RedirectMatch 301 ^/([0-9]{4})/([0-9]{2})/([0-9]{2})/(.*)$ http://villasdiani.com/$4 <ifmodule mod_rewrite.c="">RewriteEngine On RewriteBase / RewriteCond %{QUERY_STRING} (^|?|&)s2member_file_download=.+ RewriteRule .* - [E=no-gzip:1]</ifmodule> END s2Member GZIP exclusions BEGIN WPSuperCache <ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /
#If you serve pages from behind a proxy you may want to change 'RewriteCond %{HTTPS} on' to something more sensible
AddDefaultCharset UTF-8
RewriteCond %{REQUEST_URI} !^.[^/]$
RewriteCond %{REQUEST_URI} !^.//.$
RewriteCond %{REQUEST_METHOD} !POST
RewriteCond %{QUERY_STRING} !.=.*
RewriteCond %{HTTP:Cookie} !^.(comment_author_|wordpress_logged_in|wp-postpass_).$
RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP:Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP_USER_AGENT} !^.(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800). [NC]
RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC]
RewriteCond %{HTTP:Accept-Encoding} gzip
RewriteCond %{HTTPS} on
RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html.gz -f
RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html.gz" [L] RewriteCond %{REQUEST_URI} !^.[^/]$
RewriteCond %{REQUEST_URI} !^.//.$
RewriteCond %{REQUEST_METHOD} !POST
RewriteCond %{QUERY_STRING} !.=.*
RewriteCond %{HTTP:Cookie} !^.(comment_author_|wordpress_logged_in|wp-postpass_).$
RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP:Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP_USER_AGENT} !^.(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800). [NC]
RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC]
RewriteCond %{HTTP:Accept-Encoding} gzip
RewriteCond %{HTTPS} !on
RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html.gz -f
RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html.gz" [L] RewriteCond %{REQUEST_URI} !^.[^/]$
RewriteCond %{REQUEST_URI} !^.//.$
RewriteCond %{REQUEST_METHOD} !POST
RewriteCond %{QUERY_STRING} !.=.*
RewriteCond %{HTTP:Cookie} !^.(comment_author_|wordpress_logged_in|wp-postpass_).$
RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP:Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP_USER_AGENT} !^.(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800). [NC]
RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC]
RewriteCond %{HTTPS} on
RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html -f
RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index-https.html" [L] RewriteCond %{REQUEST_URI} !^.[^/]$
RewriteCond %{REQUEST_URI} !^.//.$
RewriteCond %{REQUEST_METHOD} !POST
RewriteCond %{QUERY_STRING} !.=.*
RewriteCond %{HTTP:Cookie} !^.(comment_author_|wordpress_logged_in|wp-postpass_).$
RewriteCond %{HTTP:X-Wap-Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP:Profile} !^[a-z0-9"]+ [NC]
RewriteCond %{HTTP_USER_AGENT} !^.(2.0\ MMP|240x320|400X240|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|Googlebot-Mobile|hiptop|IEMobile|KYOCERA/WX310K|LG/U990|MIDP-2.|MMEF20|MOT-V|NetFront|Newt|Nintendo\ Wii|Nitro|Nokia|Opera\ Mini|Palm|PlayStation\ Portable|portalmmm|Proxinet|ProxiNet|SHARP-TQ-GX10|SHG-i900|Small|SonyEricsson|Symbian\ OS|SymbianOS|TS21i-10|UP.Browser|UP.Link|webOS|Windows\ CE|WinWAP|YahooSeeker/M1A1-R2D2|iPhone|iPod|Android|BlackBerry9530|LG-TU915\ Obigo|LGE\ VX|webOS|Nokia5800). [NC]
RewriteCond %{HTTP_user_agent} !^(w3c\ |w3c-|acs-|alav|alca|amoi|audi|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-|dang|doco|eric|hipt|htc_|inno|ipaq|ipod|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-|lg/u|maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|palm|pana|pant|phil|play|port|prox|qwap|sage|sams|sany|sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo|teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|wap-|wapa|wapi|wapp|wapr|webc|winw|winw|xda\ |xda-).* [NC]
RewriteCond %{HTTPS} !on
RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html -f
RewriteRule ^(.*) "/wp-content/cache/supercache/%{SERVER_NAME}/$1/index.html" [L]</ifmodule> END WPSuperCache BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]</ifmodule> END WordPress0 -
Does Google crawl the pages which are generated via the site's search box queries?
For example, if I search for an 'x' item in a site's search box and if the site displays a list of results based on the query, would that page be crawled? I am asking this question because this would be a URL that is non existent on the site and hence am confused as to whether Google bots would be able to find it.
Intermediate & Advanced SEO | | pulseseo0