Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters
-
Moz crawler is crawling all pages like this:
- http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date
- http://www.xxxx.com/?product_count=100&product_order=desc&paged=1
- http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid
Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us.
Thanks.
-
The immediate solution is use your robots.txt file to block the Moz crawler from crawling URLs with parameters. Pamela.
User-agent: rogerbot
Disallow: /*?utmThose pages are coming from the bot trying to follow links to all the different ways product pages can be sorted. You'll want to insure Googlebot isn't having the same problem.
Hope that helps;
Paul
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can a page have high Google/ organic traffic but show no ranking keywords in Moz?
We have a page on our website with a higher than average number of pageviews, 85% of which came from Google organic search. When I research this page by entering the URL into the "exact page" keyword research tool, Moz says it has no ranking keywords. How can a page be earning organic traffic without ranking for any keywords?
Moz Bar | | baystatemarketing0 -
Moz is showing issues at metadata continuously even though the issues are fixed.
I crawled my website by Moz and found many metadata issues (Short meta description, Too long title, Too long URL). I fixed all of the issues. But when I recrawled my site it showed me all issues are fixed except Meta description. I thought maybe my changes are not being saved and I checked again but It seemed okay! All the changes I made it is applied. So, my meta description is okay now but Moz is appearing old meta description what was too short and detecting as an issue. I recrawled my site 4 times. Please help me with that issue. Thanks, Robin
Moz Bar | | Lobin0 -
On-Page Grader Url is inaccessible
Hi everybody. I'm trying to use on -page grader for https://www.upscaledinnerclub.com and get "Sorry, but that URL is inaccessible." Robots.txt are empty, another thread on MOZ was talking about DNS check - it's all good. So, I can't figure out why this is happening. Also I am trying the same for another website https://www.regexseo.com - the same story. Common thing is that they both are on Google App Engine. And at first i thought that was the problem. Bu then i checked this one : https://www.logitinc.com/ and it's working, even though this website is on GAE as well. None of these website have robots.txt or any differences in setup or settings. Any thoughts?
Moz Bar | | DmitriiK0 -
Suggestion: Moz Domain Authority should take disavow into account
Since Moz is trying to predict how Google ranks your site, and Google claims to take the disavow file into account, I'd like to suggest that Moz allow webmasters to upload their disavow file. I imagine this data would be useful to Moz in determining Domain Authority (they may even think of other ways to use it and might even help come to a conclusion on the great debate) and it gives a chance for sites to improve their Moz DA when they are bombarded by spammy links. I'd love to hear the community's thoughts on this idea, as well as the what Wizards of Moz have to say.
Moz Bar | | YairSpolter1 -
Duplicate page content
The MOZ crawler identifies pages as duplicate content which are not the same.
Moz Bar | | aignerart
The pages http://www.aignerart.com/abstracts-oil-painting/cicli-colora.html and http://www.aignerart.com/abstracts-oil-painting/murs-de-la-ville.html are marked duplicate but they are different paintings. Any ideas?0 -
Crawl Diagnostics - nofollow - reducing duplicate pages
Hi I'm looking at a crawl diagnostic report, I can see I have many duplicate pages, the reason for this is that when a brand filter is applied to a page. IE
Moz Bar | | chameleondm
www.mysite.com/mycategory - lets say this is the product listing page
www.mysite.com/category/mybrand - and this is the same page but with a brand filter applied
www.mysite.com/category/myotherbrand - and this is the same page but with a different brand filter applied I had intially appendeded the meta title, description and keywords with some extra content if a brand filter was applied, because the page on the whole does have different content. IE I would have a custom meta information, H1 tag and products on that page just for that specific brand.
However I am wondering if these two pages are really just competing with each other as lots of the content will be the same. Should I scrap that approach and use either nofollow on the brand filter link, or simply use a canonical. Thanks, James1 -
Moz Analytics/Reports into PDF
Hey guys The Moz Analytics is great however I don't seem to be able to download the data into a PDF. I used to be able to do this but it won't let me anymore. Its vital we can do this so we can send to our clients anybody got any ideas? Am I being blind or has this been omitted??
Moz Bar | | tempowebdesign1 -
Moz showing warnings for each dynamic link despite canonicalization?
As you can see in the attached image, Moz is showing a warning for each dynamic URL despite a rel=canonical tag. Is this by design? If so, it is frustrating seeing as it is really just the one page with many links . . . 9h5oDmr.png
Moz Bar | | BlueLinkERP0