Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters
-
Moz crawler is crawling all pages like this:
- http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date
- http://www.xxxx.com/?product_count=100&product_order=desc&paged=1
- http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid
Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us.
Thanks.
-
The immediate solution is use your robots.txt file to block the Moz crawler from crawling URLs with parameters. Pamela.
User-agent: rogerbot
Disallow: /*?utmThose pages are coming from the bot trying to follow links to all the different ways product pages can be sorted. You'll want to insure Googlebot isn't having the same problem.
Hope that helps;
Paul
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Utilizing the Moz Location in keywords vs including the location in the actual keyword phrase
I've searched the Moz blog as well as Q&A forums - but I specifically have a question about utilizing the Moz Location in keywords vs including the location in the actual keyword phrase. I've done both and get different rankings. For instance, I have "it company" with the Location set to Corpus Christi, Texas and our rank is #3. Then I also have "it company corpus christi" with the Location set to Corpus Christi, Texas and our rank is #6. Additionally, I threw in "it company corpus christi" with the Location set to National and our rank is #7. What is the best practice and what are the determining factors for the differences? Thank you in advance!
Moz Bar | | LBoxerger0 -
MozBot Finding Duplicate Pages That Aren't Duplicate
I've been reviewing the technical audits for my campaign in Moz, and noticed I had a number of duplicate content issues that I'm not really sure how to address. When I click on the links of what the duplicates are, they are all different links that have different content/images. Based on what I was seeing other's wrote in the forum, this could be because the code base is really the same between these pages, and many of these were using query parameters (I'm assuming that is why the code is almost exactly the same across these pages), so example: website.com/tags/KEYWORD1?type=KEYWORD2 is a duplicate of website.com/tags/KEYWORD3?type=KEYWORD4 I was reading that I can use that URL Parameters area in google search console, but my search console says that the googlebot isn't experiencing issues, so I wasn't sure if that was the right move. I can't do the canonicals because these pages all have different content on them, and I know duplicate content is a big SEO issue, so I really wasn't sure what my next steps should be. Thanks for the help!
Moz Bar | | amaray4030 -
How to Find Related Keywords in Moz.com
Hello, I have used the SEMRush tool and it provides related keywords with other metrics. How do I find the same using Moz.com.
Moz Bar | | Chaku0 -
Duplicate Page and Title Issues
On the last crawl, we received errors for duplicate page titles and some duplicate content pages. Here is the issue: We went through our page titles that were marked as duplicate and changed them to make sure their titles were different. However, we just received a new crawl this week and it is saying there are even more duplicate page title errors detected than before. We're wondering if this is a problem with just us or if it has been happening to other Moz users. As for the duplicate content pages, what is the best way to approach this and see what content is being looked at as a "duplicate" set?
Moz Bar | | Essential-Pest0 -
On Page Grader - We have keyword in Meta Description But does not show up
The on-page grader is not counting/picking up the keyword in the meta description. We have double checked the meta description and re-uploaded the page, and checked the source code on-line. The keyword is there in meta description, but not showing up in on-page-grader. https://www.trustdeedrealestateinvesting.com/trimarkfunding/buy-first-trust-deeds.html keyword buy first trust deeds
Moz Bar | | Manifestation0 -
403 Error on WMT but not on MOZ?
Hello, 2 days ago I found there are about 1200 of 403 errors by Google WMT when I tried to fetch my domain - Please see attached HTTP/1.1 403 Access Forbidden Cache-Control: private Content-Type: text/html ETag: "" Server: Set-Cookie: ASPSESSIONIDSSBARTSD=BEHMJHJBKJOEJEALECNNIPFH; path=/; HttpOnly X-Powered-By: Date: Tue, 18 Feb 2014 13:54:10 GMT Content-Length: 1233 <title>403 - Forbidden: Access is denied.</title> Server Error <fieldset> 403 - Forbidden: Access is denied. You do not have permission to view this directory or page using the credentials that you supplied. </fieldset> I ran a complete report using MOZ but I was shocked not see any 4xx , 5xx errors. Google: 246 of 404 errors No Google, Yahoo or Bing blocking HTTP status code: ALL 200 301 redirect: none? I have done about 2500 over 4 years. The website is losing indexed pages. I'm not sure what's going and which numbers to trust. Please help. Thank you. Adam
Moz Bar | | homs830 -
Moz toolbar Broken ?
Hi everybody, It seems the Moz toolbar isn't working properly for two days (Amsterdam Time zone GWMT +1) - we are located in the Netherlands. My colleague and me are have the same experience. Nothing is displayed anymore (no PA, mR, DA or Links) the sections with the pencil (follow and no follow) is working. Working in firefox 26.0.Does anybody has the same issue? Moz is this a global / local problem?
Moz Bar | | lveembergen0 -
Screaming Frog, Moz and other crawlers
Hi Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done. Thanks in advance for any support. Neil
Moz Bar | | mccormackmorrison0