How to resolve - Googlebot found an extremely high number of URLs
-
Hi,
We got this message from Google Webmaster “Googlebot found an extremely high number of URLs on your site”. The sample URLs provided by Google are all either noindex or have a canonical.
- http://www.myntra.com/nike-stylish-show-caps-sweaters
- http://www.myntra.com/backpacks/f-gear/f-gear-unisex-black-&-purple-calvin-backpack/162453/buy?src=tn&nav_id=541
- http://www.myntra.com/kurtas/alma/alma-women-blue-floral-printed-kurta/85178/buy?nav_id=625
Also we have specified the parameters on these URLs as representative URL in Google Webmaster - URL parameters.
Your comments on how to resolve this issue will be appreciated.
Thank You
Kaushal Thakkar
-
Hi Kaushal,
Thanks for the question.
There are a few ways to deal with this problem which are recommended by Google here. In summary, you can:
- Use parameter handling as you have done
- Add the nofollow attribute to problematic URLs
- Block problematic URLs in robots.txt
There is also a thread in the Google webmaster forums which may be useful to you:
Overall, it comes down to having a good site architecture and cutting down / removing / blocking URLs that you don't care about from a search perspective.
I hope that helps a bit!
Paddy
-
Thank you David, Its been more than 10 months since these parameters have been specified in webmaster. This and other activities like noindex and canonicals helped us to reduce the indexed URL count from 32 million to 1.2 million. As the url index reduced this warning from google stopped for 4 months. However we started receiving this message again from february 2014.
Thanks
Kaushal
-
"we have specified the parameters on these URLs as representative URL in Google Webmaster - URL parameters."
How long ago was this done? Since there are so many URL's, it may take a while for them to recrawl and index the representative URL's per your request.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Googlebot crawling AJAX website not always uses _escaped_fragment_
Hi, I started to investigate googlebot crawl log of our website, and it appears that there is no 1:1 correlation between a crawled URL with escaped_fragment and without it.
White Hat / Black Hat SEO | | yohayg
My expectation is that each time that google crawls a URL, a minute or so after, it suppose to crawl the same URL using an escaped_fragment For example:
Googlebot crawl log for https://my_web_site/some_slug Results:
Googlebot crawled this URL 17 times in July: http://i.imgur.com/sA141O0.jpg Googlebot crawled this URL additional 3 crawls using the escaped_fragment: http://i.imgur.com/sOQjyPU.jpg Do you have any idea if this behavior is normal? Thanks, Yohay sOQjyPU.jpg sA141O0.jpg0 -
Permanently Moving Few High Ranking Pages from One Domain to Another
We are planning to move few high ranking pages permanently (301 Permanent Redirection) to another domain, Currently these pages are getting good traffic from organic search and ranking on top positions in Google search engine result pages. We have few questions in our mind right now, It would be a great help if anyone can answer following questions; Is it possible to move few pages from one domain to another by using 301 Redirection in .htaccess file? Will it have any negative impact on our website's current search engine performance? Will it be considered as a legitimate SEO practice by Google Search Engine? Will Google understand that these pages have been moved permanently to another domain and start showing URL's from the new domain on the same positions where they were ranking before moving to new location?
White Hat / Black Hat SEO | | tigersohelll0 -
Where is the rule here that forces any home page URLs that include several versions of explicit page name references, such as default.htm or index.html, to redirect to the canonical homepage?
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)/(1)?$ /search/$1/sort-$2 [R=301]
White Hat / Black Hat SEO | | esiow2013
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$5&sort=$2 [L]
RewriteRule ^search/([a-zA-Z0-9,-]+)/(1)?$ /search/$1 [R=301]
RewriteRule ^search/([a-zA-Z0-9,-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$4 [L]
RewriteRule ^([r|R])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=1 [L]
RewriteRule ^([s|S])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=2 [L]
RewriteRule ^reset-password$ index.php?cmd=retrievePassword [L]
RewriteRule ^reset-password/([0-9]+)/([a-zA-Z0-9]+)$ index.php?cmd=retrievePassword&act=retrievePasswordConfirm&code=$1&key=$2 [L]
RewriteRule ^map-page.php$ index.php?cmd=mapPage [L]
RewriteRule ^advertise.php$ index.php?cmd=advertiseWithUs [L]
RewriteRule ^my-account.php$ index.php?cmd=protectedMyAccount [L]
RewriteRule ^bookmarked-properties(/((R|S)-([0-9]+)))?$ index.php?cmd=protectedBookmarkedProperties&id=$2 [L]
RewriteRule ^bookmarked-properties/([0-9]+)$ index.php?cmd=protectedBookmarkedProperties&page=$1 [L]
RewriteRule ^bookmarked-properties/delete/([0-9,]+)$ index.php?cmd=protectedBookmarkedProperties&act=protectedBookmarkedProperties_delete&id=$1 [L]
RewriteRule ^register.php$ index.php?cmd=register [L]
RewriteRule ^about-us.php$ index.php?cmd=aboutUs [L]
RewriteRule ^news(/([0-9]+)(.*))?$ index.php?cmd=news&id=$2&title=$3 [L]
RewriteRule ^registration/confirm/key=([a-zA-Z0-9]{150})/code=([a-zA-Z0-9]{5})$ index.php?cmd=register&act=registerConfirm&key=$1&code=$2 [L]
RewriteRule ^terms-and-conditions.php$ index.php?cmd=termsAndConditions [L]
RewriteRule ^privacy-policy.php$ index.php?cmd=privacyPolicy [L]
RewriteRule ^contact-us.php$ index.php?cmd=contactUs [L]
RewriteRule ^sitemap.php$ index.php?cmd=siteMap [L]
RewriteRule ^members/login.php$ login.php [L]
RewriteRule ^rss$ index.php?cmd=rss [L]
RewriteRule ^resources.php$ index.php?cmd=links [L]
RewriteRule ^links.php$ /resources.php [R=301]
RewriteRule ^resources2.php$ index.php?cmd=links2 [L]
RewriteRule ^results.php$ index.php?cmd=results [L]
RewriteRule ^residential-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&page=$3 [L]
RewriteRule ^commercial-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=2&page=$3 [L]
RewriteRule ^short-term-rentals((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&str=1&page=$3 [L]
RewriteRule ^buy((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=2&page=$3 [L]
RewriteRule ^rent((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=1&page=$3 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&sort=$2&page=$5 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&sort=$4&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&sort=$5&page=$8 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&page=$6 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=2&page=$4 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=2&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&page=$5 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&page=$5 [L]
RewriteRule ^rent/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=2&page=$4 [L]0 -
Solved PayDay hack - but SERPs show URLs - what should I do?
We had the PayDay hack - and solved it completely. The problem is - the SERPs have over 3,000 URLs pointing to 404 on our website all of which have urls that are like this: <cite>www.onssi.com/2012/2/post1639/payday-loan-companies-us</cite> What should I do? Should I disavow every one of the 3,000? No Follow?
White Hat / Black Hat SEO | | Ocularis0 -
Google Sitemaps & punishment for bad URLS?
Hoping y'all have some input here. This is along story, but I'll boil it down: Site X bought the url of Site Y. 301 redirects were added to direct traffic (and help transfer linkjuice) from urls in Site X to relevant urls in Site Y, but 2 days before a "change of address" notice was submitted in Google Webmaster Tools, an auto-generating sitemap somehow applied urls from Site Y to the sitemap of Site X, so essentially the sitemap contained urls that were not the url of Site X. Is there any documentation out there that Google would punish Site X for having essentially unrelated urls in its sitemap by downgrading organic search rankings because it may view that mistake as black hat (or otherwise evil) tactics? I suspect this because the site continues to rank well organically in Yahoo & Bing, yet is nonexistent on Google suddenly. Thoughts?
White Hat / Black Hat SEO | | RUNNERagency0 -
Google Sitemaps & punishment for bad URLS?
Hoping y'all have some input here. This is along story, but I'll boil it down: Site X bought the url of Site Y. 301 redirects were added to direct traffic (and help transfer linkjuice) from urls in Site X to relevant urls in Site Y, but 2 days before a "change of address" notice was submitted in Google Webmaster Tools, an auto-generating sitemap somehow applied urls from Site Y to the sitemap of Site X, so essentially the sitemap contained urls that were not the url of Site X. Is there any documentation out there that Google would punish Site X for having essentially unrelated urls in its sitemap by downgrading organic search rankings because it may view that mistake as black hat (or otherwise evil) tactics? I suspect this because the site continues to rank well organically in Yahoo & Bing, yet is nonexistent on Google suddenly. Thoughts?
White Hat / Black Hat SEO | | RUNNERagency0 -
Low Quality Highly Relevant backlinks, should we get them?
I see a lot of opportunity to get lower quality, but highly relevant backlinks, should we try to get these? I'll give you an example, lets say we have an asphalt paving company ( not a lot of authority blogs out there, that we can find yet) We found this one http://www.wolfpaving.com/blog/ - DA of 27 and PA 29 should we go after links like this. I would actually like to know about sites with less authority than this one, I would probably go for this one without question. So Should we go after worse DA and PA but still legitimate looking sites and highly relevant?
White Hat / Black Hat SEO | | RonMedlin0 -
Stuffing keywords into URLs
The following site ranks #1 in Google for almost every key phrase in their URL path for almost every page on their site. Example: themarketinganalysts.com/en/pages/medical-translation-interpretation-pharmaceutical-equipment-specifications-medical-literature-hippa/ The last folder in this URL uses 9 keywords and I've seen as many as 18 on the same site. Curious: every page is a "default.html" under one of these kinds of folders (so much architecture?). Question: How much does stuffing keywords into URL paths affect ranking? If it has an effect, will Google eventually ferret it out and penalize it?
White Hat / Black Hat SEO | | PaulKMia0