Seomoz crawling filtered pages
-
Hi,
I just checked an seo campaign we started last week, so I opened seomoz to see the crawl diagnostics.
Lot's of duplicate content & duplicate titles showing up, but that's because Rogerbot is crawling all of the filtered pages as well. How do I exclude these pages from being crawled?
/product/brand-x/3969?order=brand&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=DESC&page=10
/product/brand-x/3969?order=popular&sortorder=DESC&page=11 -
So if the site has a structure like this:
and the filter on this page has several options like /?order= , ?brand=, .... I have to rel-canonical them al to www.xyz.com/overview
-
I'd rel-canonical if you can, as theres still nothing stopping links to them being indexed. It might stop Rodger/Google from crawling them, but the potential indexation issues won't go away. Otherwise perhaps no-index them.
I'd usualy go as far to do re-prev and rel-next for the paginated searches as well.
-
Rel-canonical will be an immense job, would it be ok using robots.txt like this
disallow: /?order=
disallow: /?sortorder=
...
?
-
Last I checked, Rodger obays robot.txt and meta-robot commands as he trys to simulate what google would crawl. If he can crawl those pages, google probably is as well.
I think if you rel-canonical them properly, or no-idex them properly e.t.c. It should show as a normal page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz can't crawl my site
Moz is being blocked from crawling the following site - https://www.cleanchain.com. When looking at Robot.txt, the following is disallowing access but don't know whether this is preventing Moz from crawling too? User-agent: *
Moz Pro | | danhart2020
Disallow: /adeci/
Disallow: /core/
Disallow: /connectors/
Disallow: /assets/components/ Could something else be preventing the crawl?0 -
Problem crawling a website with age verification page.
Hy every1, Need your help very urgent. I need to crawl a website that first has a page where you need to put your age for verification and after that you are redirected to the website. My problem is that SEOmoz, crawls only that first page, not the whole website. How can I crawl the whole website?, do you need me to upload a link to the website? Thank you very much Catalin
Moz Pro | | catalinmoraru0 -
How to resolve Duplicate Content crawl errors for Magento Login Page
I am using the Magento shopping cart, and 99% of my duplicate content errors come from the login page. The URL looks like: http://www.site.com/customer/account/login/referer/aHR0cDovL3d3dy5tbW1zcGVjaW9zYS5jb20vcmV2aWV3L3Byb2R1Y3QvbGlzdC9pZC8xOTYvY2F0ZWdvcnkvNC8jcmV2aWV3LWZvcm0%2C/ Or, the same url but with the long string different from the one above. This link is available at the top of every page in my site, but I have made sure to add "rel=nofollow" as an attribute to the link in every case (it is done easily by modifying the header links template). Is there something else I should be doing? Do I need to try to add canonical to the login page? If so, does anyone know how to do it using XML?
Moz Pro | | kdl01 -
How do you use SEOmoz
I'm a newbie here. I've got a week left on my free trail. I really like SEOmoz, I might even love it a bit. 🙂 I'm trying to justify the cost. I usually don't spend money on my Internet Marketing. I've done it through finally buying some tools to help me get there. Massive change in the last year. I'm a long ways from getting rich YET though. 🙂 I've gone from $30 a month to $1000 a month in a year. So I think I'm on my way now. I'm trying to figure out why I should continue my subscription and actually start paying for it. If I were a developer I can certainly see why but the cost would so go way up. 🙂 I'm in the lowest level package with only five websites. I have 10 I'm working on. I know that's probably five to many at one time. I'm having a little trouble justifying the cost at this time. At my income level $100 a month is significant. I can see the value of seeing where my competition has backlinks. Sometimes my competition is Cnet. haha Well, I can beat them on a specific page at times. Hmm, can I search the URL list for links? Perhaps i can if I export it. That didn't occur to me. Now that could be of use if I can find my keywords. If their link is on Amazon though, I'm going to have trouble getting a link there. 🙂 Maybe I should work on less competitive links but I made good money last month with one page that was beating them. WAS beating them. How do you use SEOmoz? How do you justify the cost, that is offset the expense. If its not making me money i don't see the value. I can get a lot of this information for free but it sure is easier to get it here. Which saves me time. And the other tool i use isn't that good and not always reliable and slow. I've got a week left on my free trail. I had decided to let it go but now I'm reconsidering. $100 is a lot for me and I need to spend more on SEO outsourcing. I wonder if I wouldn't be better off dropping it for a while and then coming back. This certainly isn't mean to be a critical post at all. I'm looking for reasons to stay and make better use of it. Thanks for the thoughts, Rusty
Moz Pro | | RustyF0 -
Crawl test tool from SEOmoz - which URLs does it actually crawl?
I am using for the first time the crawl test tool from SEOmoz and I do not really understand which URLs the tool is going to crawl. First, it says "enter any subdomain" --> why can´t I do the crawl for the root domain? Second it says "we'll crawl up to 3,000 linked-to pages" --> does that mean that the tool crawls all internal links that it can find on the given domain? Thanks for your help!
Moz Pro | | Elke.GetApp0 -
Excluding parameters from seomoz crawl?
I'm getting a ton of duplicate content errors because almost all of my pages feature a "print this page" link that adds the parameter "printable=Y" to the URL and displays a plain text version of the same page. Is there any way to exclude these pages from the crawl results?
Moz Pro | | AmericanOutlets0 -
Campaign Not Crawling
I set up my first 5 campaigns and one is not crawling beyond one-page. It's been over 48 hours. This site has nearly 3.500 pages the others much less, however, this shouldn't make any difference. I searched for the problem and couldn't find it so I hope this question isn't redundant. Comments and advice would be appreciated.
Moz Pro | | JavaManOne0 -
SEOMoz's Crawl Diagnostics showing an error where the Title is missing on our Sitemap.xml file?
Hi Everyone, I'm working on our website Sky Candle and I've been running it as a campaign in SEOmoz. I've corrected a few errors we had with the site previously, but today it's recrawled and found a new error which is a missing Title tag on the sitemap.xml file. Is this a little glitch in the SEOmoz system? Or do I need to add a page title and meta description to my XML file. http://www.skycandle.co.uk/sitemap.xml Any help would be greatly appreciated. I didn't think I'd need to add this. Kind Regards Lewis
Moz Pro | | LewisSellers0