Seomoz crawling filtered pages
-
Hi,
I just checked an seo campaign we started last week, so I opened seomoz to see the crawl diagnostics.
Lot's of duplicate content & duplicate titles showing up, but that's because Rogerbot is crawling all of the filtered pages as well. How do I exclude these pages from being crawled?
/product/brand-x/3969?order=brand&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=DESC&page=10
/product/brand-x/3969?order=popular&sortorder=DESC&page=11 -
So if the site has a structure like this:
and the filter on this page has several options like /?order= , ?brand=, .... I have to rel-canonical them al to www.xyz.com/overview
-
I'd rel-canonical if you can, as theres still nothing stopping links to them being indexed. It might stop Rodger/Google from crawling them, but the potential indexation issues won't go away. Otherwise perhaps no-index them.
I'd usualy go as far to do re-prev and rel-next for the paginated searches as well.
-
Rel-canonical will be an immense job, would it be ok using robots.txt like this
disallow: /?order=
disallow: /?sortorder=
...
?
-
Last I checked, Rodger obays robot.txt and meta-robot commands as he trys to simulate what google would crawl. If he can crawl those pages, google probably is as well.
I think if you rel-canonical them properly, or no-idex them properly e.t.c. It should show as a normal page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site not crawl?
hi all me allow rogerbot in robots.txt but rogerbot can't crawl my site my site: toska-co.ir
Moz Pro | | jahanidawodi0 -
How to know exactly which page links to a 404 page on my website?
Hi Moz users, Sometimes I get 404 crawl errors using Moz Pro and when my website has a few dozen pages it is hard for me to find the original page that links to a 404 page. Is there a way to find this automatically using Moz or do I have to look for it manually? I just need to find the original link and delete it to fix my 404 issue. Please let me know thank you for you help. -Marc
Moz Pro | | marcandre0 -
Ajax4SEO and rogerbot crawling
Has anyone had any experience with seo4ajax.com and moz? The idea is that it points a bot to a html version of an ajax page (sounds good) without the need for ugly urls. However, I don't know how this will work with rogerbot and whether moz can crawl this. There's a section to add in specific user agents and I've added "rogerbot". Does anyone know if this will work or not? Otherwise, it's going to create some complications. I can't currently check as the site is in development and the dev version is noindexed currently. Thanks!
Moz Pro | | LeahHutcheon0 -
How to make a Crawl Report readable?
Hi! I am trying to find out how I make my CSV report neat, so I can interprete the report. I know have a CSV report with just numbers and text all in one column. I tried the button text to columns but that doesn't work because when I do that at column A it overwrites column B in which I have the same problem! Thanks
Moz Pro | | HetCommunicatielokaal0 -
Third crawl of my sites back to 250 pages
Hi all, I've been waiting some days for the third crawl of my sites, but SEOMOZ only crawled 277 pages. The next phrase appeared on my crawl report: Pages Crawled: 277 | Limit: 250 My last 2 crawls were of about 10K limit. Any idea? Kind regards, Simon.
Moz Pro | | Aureka0 -
SEOMOZ can we trust you?
I was thinking today what are the chances that SEOMOZ can look into my campaigns and see what keywords I am targeting, strategies etc. and then sell that information to others? There has to be some type of clause when you sign up correct? But who knows because I'm not sure. Having Rand and Matt Curts talking via twitter to each other kind of concerns me. Not that I'm worried that Google will know my strategies because we don't do any black-hat linking and anything shady but how much power does SEOMOZ really have and can they leak any information to other companies or Google? Scary stuff that's why I don't put all my eggs into one basket, I spread them out. Thoughts? Thanks in advance - Brett Shaffer
Moz Pro | | EmpireSEO1 -
Campaign Crawl Report
Hello, Just a quicky, is there anyway I can do a crawl report for something in a campaign so I can compare the changes? I know you can do a separate crawl test, but it wont show the differences,and the next crawl date isnt untill the 28th.
Moz Pro | | Prestige-SEO0 -
SEOMoz Keyword Limit
I am trying to add keywords to my SEOMoz campaign and it says I have reached my 300 KW limit eventhough I have only added 1 word! Any suggestions? uwt2V
Moz Pro | | theLotter0