Googlebot found an extremely high number of URLs on your site:
-
Website: www.gobol.in
Although I have no indexed my search pages by adding /catalogsearch in robots.txt, still we are getting same error again and again
Here's a list of sample URLs with potential problems.
- http://www.gobol.in/catalogsearch/result/index/?category=&mobile_feature=4575_4578&q=panasonic+NR-BU303LH1H+REFRIGERATOR+296+L+GREY&special_price=32%2C456&x=0&y=0
- http://www.gobol.in/mobile-and-accessories/mobiles-and-brands.html?manufacturer=4753_3355_455_4435_4720_3407_2412_4728_4784_4790_2010_4789_4376_2469&operating_system_mobile=4612
Please help
-
Thanks Dirk
-
Hi Ravi,
Did you use the robots.txt tester in Webmastertools - your robots.txt mentions to block /catalogsearch - if you want to block the entire directory I think you have to use
Disallow: /catalogsearch/
On top of that /catalogsearch is only blocked for Googlebot-image - not for the "normal" Googlebot - if you want to have it blocked for all bots you have to move it to the beginning of the file. Idem for /admin => should be /admin/ and in the beginning of your robots.txt file
rgds,
Dirk
-
Thanks Martijn
-
Hi Ravi,
This message you got from Google is more a warning then an error, that is good to keep in mind. As there is no problem in theory with having a lot of URLs on your site if the quality is high enough. Therefore the first URL shouldn't cause any troubles, as like you mention, it's included in the robots.txt file.
The second URL is a bit less nice but shouldn't also be trouble, as it's making sure that the URL parameters are working. Also there is a canonical URL on all these filter pages to make sure the pages are originally linked back to the category page. It could be worth it to check the URL parameters within Google Webmaster Tools to make sure Google is aware of all the changes in content for different parameters.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Consolidating a Large Site with Duplicate Content
I will be restructuring a large website for an OEM. They provide products & services for multiple industries, and the product/service offering is identical across all industries. I was looking at the site structure and ran a crawl test, and learned they have a LOT of duplicate content out there because of the way they set up their website. They have a page in the navigation for “solution”, aka what industry you are in. Once that is selected, you are taken to a landing page, and from there, given many options to explore products, read blogs, learn about the business, and contact them. The main navigation is removed. The URL structure is set up with folders, so no matter what you select after you go to your industry, the URL will be “domain.com/industry/next-page”. The product offerings, blogs available, and contact us pages do not vary by industry, so the content that can be found on “domain.com/industry-1/product-1” is identical to the content found on “domain.com/industry-2/product-1” and so-on and so-forth. This is a large site with a fair amount of traffic because it’s a pretty substantial OEM. Most of their content, however, is competing with itself because most of the pages on their website have duplicate content. I won’t begin my work until I can dive in to their GA and have more in-depth conversations with them about what kind of activity they’re tracking and why they set up the website this way. However, I don’t know how strategic they were in this set up and I don’t think they were aware that they had duplicate content. My first thought would be to work towards consolidating the way their site is set up, so we don’t spread the link-equity of “product-1” content, and direct all industries to one page, and track conversion paths a different way. However, I’ve never dealt with a site structure of this magnitude and don’t want to risk messing up their domain authority, missing redirect or URL mapping opportunities, or ruin the fact that their site is still performing well, even though multiple pages have the same content (most of which have high page authority and search visibility). I was curious if anyone has dealt with this before and if they have any recommendations for tackling something like this?
On-Page Optimization | | cassy_rich0 -
Duplicate URL errors when URL's are unique
Hi All, I'm running through MOZ analytics site crawl report and it is showing numerous duplicate URL errors, but the URLs appear to be unique. I see that the majority of the URL's are the same, but shouldn't the different brands make them unique to one another? http://www.sierratradingpost.com/clearance~1/clothing~d~5/tech-couture~b~33328/ http://www.sierratradingpost.com/clearance~1/clothing~d~5/zobha~b~3072/ Any ideas as to why these would be shown as duplicate URL errors?
On-Page Optimization | | STP_SEO0 -
Strange URL resulting a page
Hi, my friend has asked me to take a look at his site. I only know the basics of SEO so I'm learning along the way. He has some duplicate title errors showing in Moz, resulting to this page: https://www.domainname.com/about/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers This URL shows the 'About' page. I have tonnes of pages like this showing with really long URLs that result an actual page. Has anyone seen something like this before? I don't have a clue how this is showing the about page Any advice is greatly appreciated. Thanks James
On-Page Optimization | | Craze_Media0 -
Which pages on my site should I back link to
The majority of the back links I have been creating link directly to our home page and to the store page. Is this the best approach or should I be trying to spread the links throughout our site to include product categories and subcategories etc?
On-Page Optimization | | Hardley0 -
Site Redesign: Potential Problems?
We're looking at a cleanup of our site and making our navigation menu simpler and more visual. The problem is there's a massive fear that we will incur some sort of SEO hit at a time when we can ill afford one. The bulk of the changes are around some tertiary pages that have some SEO value but are not primary targets or traffic sources. The pages with content aren't moving URLs or changing, nor is the core content of the site, just the template. The new template, I feel, is easier to navigate and easier to spider (menus are flyout divs with images). So are there any tips or tricks to changing the layout of your site without hurting SEO? Anything we need to avoid?
On-Page Optimization | | Highland0 -
On my site, www.myagingfolks.com, only a small number of my pages appear to be indexed by google or yahoo. Is that due to not having an XML sitemap, keywords, or some other problem?
On my site, www.myagingfolks.com, only a small number of my pages appear to be indexed by google or yahoo. I have thousands of pages! Is that due to not having an XML sitemap, keywords, or some other problem?
On-Page Optimization | | Jordanrg0 -
Best URL Structure For Products That Are The Same
I know that the url structure is very important for seo preferably using the keyword. But is it okay to have the same url with the product number at the end ? Each of our products have a name with a product number. Or will this cause to many similar urls? or if the folder is the name of the product that needs to be optimized, can the page just be called the product number? Example: Say you have a 20 different product lines and they are all catagorized in the appropriate folders, and need to be optimized for the actual product name. XXX (folder name ) WWW-PR-123 WWW-PR-1234 WWW-PR-12345 WWW-PR-123456 what would be the best url structure? Can they have the same begining? The product name? something like: www.example.com/xxx/www-pr-123.php www.example.com/xxx/www-pr-1234.php or www.example.com/xxx/pr-123.php www.example.com/xxx/pr-1234.php
On-Page Optimization | | hfranz0 -
For Service Sites, having a phone number in meta description ALWAYS a good idea?
If you have a site that offers a same day service, e.g. plumbing, electrical, computer support: do you think it would always be a good idea to stick the phone # in the meta description?
On-Page Optimization | | ilyaelbert1